Virtual Window

ABSTRACT

A virtual window system, the system comprising a light field camera device and a light field display device arranged back-to-back, the light field camera device configured to capture an input light field video stream, the light field display device configured to display an output light field video stream based on the input light field video stream.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of application Ser. No. 14/269,071,filed May 2, 2014, which is a continuation of application Ser. No.13/567,010, filed Aug. 4, 2012, now U.S. Pat. No. 8,754,829, both ofwhich are incorporated herein by reference in their entirety.

FIELD OF THE INVENTION

The present invention relates to high-fidelity light field displays,cameras, and two-way displays.

BACKGROUND OF THE INVENTION

A 7D light field (or plenoptic function [Adelson91]) defines thespectral radiance of every ray passing through every point in a volumeof space over time, and therefore contains every possible view withinthat volume. A 6D light field defines the spectral radiance of every raypassing through a given surface over time, i.e. it represents a slicethrough a 7D light field.

Typically, only rays passing through the surface in one direction are ofinterest, e.g. rays emitted by a volume bounded by the surface. The 6Dlight field at the boundary can be used to extrapolate the 7D lightfield of the surrounding space, and this provides the basis for a lightfield display. The extrapolation is performed by rays emitted by thedisplay as they propagate through space.

Although an optical light field is continuous, for practicalmanipulation it is band-limited and sampled, i.e. at a discrete set ofpoints on the bounding surface and for a discrete set of ray directions.

The ultimate purpose of a light field display, in the present context,is to reconstruct a continuous optical light field from an arbitrarydiscrete light field with sufficient fidelity that the display appearsindistinguishable from a window onto the original physical scene fromwhich the discrete light field was sampled, i.e. all real-world depthcues are present. A viewer sees a different view from each eye; is ableto fixate and focus on objects in the virtual scene at their properdepth; and experiences smooth motion parallax when moving relative tothe display.

The ultimate purpose of a light field camera, in the present context, isto capture a discrete light field of an arbitrary physical scene withsufficient fidelity that the discrete light field, when displayed by ahigh-fidelity light field display, appears indistinguishable from awindow onto the original scene.

Existing glasses-free three-dimensional (3D) displays fall into threebroad categories [Benzie07, Connor11]: autostereoscopic, volumetric, andholographic. An autostereoscopic display provides the viewer (ormultiple viewers) with a stereo pair of 2D images of the scene, eitherwithin a single viewing zone or within multiple viewing zones across theviewing field, and may utilise head tracking to align the viewing zonewith the viewer. A volumetric display generates a real 3D image of thescene within the volume of the display, either by rapidly sweeping a 0D,1D or 2D array of light emitters through the volume, or by directlyemitting light from a semi-transparent voxel array. A holographicdisplay uses diffraction to recreate the wavefronts of light emitted bythe original scene [Yaras10].

Volumetric and holographic displays both reconstruct nominally correctoptical light fields, i.e. they generate wide-field wavefronts withcorrect centers of curvature. However, volumetric displays suffer fromtwo major drawbacks: the reconstructed scene is confined to the volumeof the display, and the entire scene is semi-transparent (making itunsuitable for display applications that demand realism). Practicalholographic displays suffer from limited size and resolution, andtypically only support horizontal parallax in current implementations[Schwerdtner06, Yaras10, Barabas11].

Typical multiview autostereoscopic displays provide a limited number ofviews, so don't support motion parallax. So-called ‘holoform’autostereoscopic displays [Balogh06, Benzie07, Urey11] provide a largernumber of views (e.g. 10-50), so provide a semblance of (typicallyhorizontal-only) motion parallax. However, they do not reconstruct evennominally correct optical light fields.

SUMMARY OF THE INVENTION

In a first aspect, the present invention provides a light field displaydevice comprising an array of light field display elements populating adisplay surface, each display element comprising: a beam generator forgenerating an output beam of light; a radiance modulator for modulatingthe radiance of the beam over time; a focus modulator for modulating thefocus of the beam over time; and a scanner for scanning the beam acrossa two-dimensional angular field.

Persistence of vision, combined with a sufficiently rapid scan rate,allows a viewer to perceive the scanned beams as a continuous opticallight field.

The appropriate focus of each beam ensures that the viewer perceivessmooth parallax both between and within beams, and experiencesconsistent vergence and accommodation cues.

The light field display device optionally further comprises at least oneactuator for oscillating the display surface between at least twopositions. The oscillation may be resonant.

Oscillating the display surface allows incomplete coverage of thedisplay surface by the exit pupils of the display elements to beimproved.

The scanner is optionally configured to scan an input beam across thetwo-dimensional angular field, and each display element optionallyfurther comprises: an input focus modulator for modulating the focus ofthe input beam; a radiance sensor for sensing the radiance of the inputbeam; and a radiance sampler for sampling the radiance of the inputbeam.

This allows the light field display to also function as a light fieldcamera.

In a second aspect, the present invention provides a method fordisplaying a light field, the method comprising, for each of a set ofpositions on a display surface, the steps of: generating an output beamof light; modulating the radiance of the beam over time; modulating thefocus of the beam over time; and scanning the beam across atwo-dimensional angular field.

The radiance of the beam may be modulated in accordance with to aspecified radiance value corresponding to the position on the displaysurface and the instantaneous direction of the scanned beam within theangular field.

The specified radiance value may be retrieved from a discreterepresentation of the desired output light field. The discrete lightfield may have been received from a light field camera device.

The focus of the beam may be modulated in accordance with a specifieddepth value corresponding to the position on the display surface and theinstantaneous direction of the scanned beam within the angular field.

The specified depth may be a scene depth or a fixation depth of a viewerof the surface.

The method may include tracking the face, eyes and gaze of the viewer todetermine a relevant scene depth or the viewer's fixation depth.

The method optionally further comprises oscillating the display surfacebetween at least two positions.

The scanning step optionally further comprises scanning an input beamacross the two-dimensional angular field, and the method optionallyfurther comprises the steps of: modulating the focus of the input beamover time; sensing the radiance of the input beam over time; andsampling the radiance of the input beam at discrete times.

The sampled radiance may be stored in a discrete light field. Thediscrete light field may be transmitted to a light field display devicefor display.

DRAWINGS Figures

FIG. 1A shows a representative ray of a continuous 6D light field,traversing the boundary of a volume of interest.

FIG. 1B shows a class diagram for a sampled, i.e. discrete, 6D lightfield.

FIG. 2A shows a light sensor array sampling ray direction for aparticular ray position.

FIG. 2B shows an array of lenses sampling ray position at the lightfield boundary.

FIG. 3A shows the combined effect of the spatial extent of the lightsensor and the aperture of the lens to effect 4D low-pass filtering.

FIG. 3B shows the sampling beam of FIG. 3A focused at a point in objectspace using a lens with higher power.

FIG. 4A shows a light emitter array reconstructing ray direction for aparticular ray position.

FIG. 4B shows an array of lenses reconstructing ray position at thelight field boundary.

FIG. 5A shows the combined effect of the spatial extent of the lightemitter and the aperture of the lens to effect 4D low-pass filtering.

FIG. 5B shows the reconstruction beam of FIG. 5A focused from a virtualobject point using a lens with lower power.

FIG. 6A shows matched sampling (left) and reconstruction (right) beams,corresponding to FIGS. 3A and 5A.

FIG. 6B shows matched sampling (left) and reconstruction (right) beamsfocused at/from an object point, corresponding to FIGS. 3B and 5B.

FIG. 7A shows wavefronts emitted from an ideal light field display.

FIG. 7B shows wavefronts emitted from a multi-element light fielddisplay.

FIG. 8A shows wavefronts captured by an ideal light field display.

FIG. 8B shows wavefronts captured by a multi-element light fielddisplay.

FIG. 9A shows the eye of a viewer located in the reconstructed lightfield of a virtual point source, with the eye focused at the pointsource.

FIG. 9B shows the eye focused at a closer point than the virtual pointsource.

FIG. 9C shows the light field display of FIGS. 9A and 9B emitting thelight field of a point source coinciding with the translated objectpoint of FIG. 9B.

FIG. 10A shows a viewer gazing at a light field display emitting a lightfield corresponding to a virtual scene consisting of several objects.

FIG. 10B shows the location of one of the eyes used to determine aviewing direction through each display element, and thus, for eachviewing direction, an intersection point with a scene object.

FIG. 10C shows the gaze direction of each of the viewer's two eyes usedto estimate their fixation point.

FIG. 10D shows the plane of focus of one of the eyes, estimated from thedepth of the fixation point, and, for each viewing direction, anintersection point with the plane of focus.

FIG. 11 shows a pair of two-way light field displays connected via anetwork.

FIG. 12 shows a light field camera and a light field display connectedvia a network.

FIG. 13A shows a schematic diagram of an array-based two-way light fielddisplay element with a liquid crystal lens in a passive state.

FIG. 13B shows a schematic diagram of the array-based two-way lightfield display element with the liquid crystal lens in an active state.

FIG. 14A shows a schematic diagram of an array-based two-way light fielddisplay element with dual liquid crystal lenses, with the first lensactive.

FIG. 14B shows a schematic diagram of the array-based two-way lightfield display element with dual liquid crystal lenses, with the secondlens active.

FIG. 15 shows a block diagram of a scanning light field display element.

FIG. 16 shows a block diagram of an RGB laser beam generator withmultiple intensity modulators.

FIG. 17 shows a block diagram of a scanning light field camera element.

FIG. 18 shows a block diagram of a scanning two-way light field displayelement.

FIG. 19A shows a plan view of an optical design for the scanning two-waylight field display element, with output rays.

FIG. 19B shows a front elevation of the optical design for the scanningtwo-way light field display element, with output rays.

FIG. 20 shows the angular reconstruction filter of FIG. 19A implementedusing an array of lenslets.

FIG. 21A shows a plan view of the optical design for the scanningtwo-way light field display element, with input beams.

FIG. 21B shows a front elevation of the optical design for the scanningtwo-way light field display element, with input beams.

FIG. 22A shows a plan view of a biaxial MEMS scanner with an elevatedmirror.

FIG. 22B shows a cross-sectional elevation of the biaxial MEMS scannerwith an elevated mirror.

FIG. 23A shows the scanning mirror of FIG. 21A scanning a stationarybeam corresponding to a fixed point source across a linear photodetectorarray.

FIG. 23B shows the photodetector array consisting of an analogphotodetector array coupled with an analog shift register.

FIG. 24 shows a block diagram of a multi-element light field display.

FIG. 25A shows a plan view of an optical design for a two-way lightfield display, 5 elements wide, with output rays.

FIG. 25B shows a front elevation of the optical design for the two-waylight field display, consisting of 10 rows of 5 elements, with outputbeams.

FIG. 25C shows a front elevation of the optical design for the two-waylight field display, consisting of 5 rows of 10 rotated elements, withoutput beams.

FIG. 26 shows a plan view of one row of the two-way light field display,rotated as shown in FIG. 25B, with each element generating a beamcorresponding to a single point source behind the display.

FIG. 27 shows a plan view of one row of the two-way light field display,rotated as shown in FIG. 25C, with each element generating a beamcorresponding to a single point source behind the display.

FIG. 28 shows a plan view of one row of the two-way light field display,rotated as shown in FIG. 25B, with each element generating a beamcorresponding to a single point source in front of the display.

FIG. 29 shows a plan view of one row of the two-way light field display,rotated as shown in FIG. 25C, with each element generating a beamcorresponding to a single point source in front of the display.

FIG. 30 shows a block diagram of a multi-element light field camera.

FIG. 31A shows a plan view of the optical design for a two-way lightfield display, 5 elements wide, with input beams.

FIG. 31B shows a front elevation of the optical design for the two-waylight field display, consisting of 10 rows of 5 elements, with inputbeams.

FIG. 31C shows a front elevation of the optical design for the two-waylight field display, consisting of 5 rows of 10 rotated elements, withinput beams.

FIG. 32 shows a plan view of one row of the two-way light field display,rotated as shown in FIG. 31B, with each element capturing a beamcorresponding to a single point source in front of the display.

FIG. 33 shows a plan view of one row of the two-way light field display,rotated as shown in FIG. 31C, with each element capturing a beamcorresponding to a single point source in front of the display.

FIG. 34A shows a cross-sectional side elevation of an oscillatingtwo-way light field display.

FIG. 34B shows a cross-sectional side elevation of the oscillatingtwo-way light field display, two display panels high.

FIG. 34C shows a cross-sectional back elevation of the oscillatingtwo-way light field display.

FIG. 34D shows a cross-sectional back elevation of the oscillatingtwo-way light field display, two display panels high and wide.

FIG. 35A shows a graph of vertical offset versus time for theoscillating display when directly driven.

FIG. 35B shows a graph of vertical offset versus time for theoscillating display when resonantly driven.

FIG. 36 shows an activity diagram for controlling the focus of a lightfield camera according to the viewers gaze.

FIG. 37 shows an activity diagram for controlling the focus of a lightfield camera according to the viewers fixation point.

FIG. 38 shows an activity diagram for displaying a light field streamfrom a light field camera.

FIG. 39 shows an activity diagram for displaying a captured light field.

FIG. 40 shows an activity diagram for displaying a synthetic lightfield.

FIG. 41 shows a block diagram of a two-way light field displaycontroller.

FIG. 42A shows eye-directed fields of display elements of a light fielddisplay.

FIG. 42B shows the foveal field of an eye on a light field display.

FIG. 43 shows a block diagram of a two-way light field displaycontroller optimised for viewer-specific operation.

DRAWINGS Reference Numerals

-   100 Ray of light field.-   102 Light field boundary.-   104 Ray intersection point with light field boundary.-   110 Light field video.-   112 Temporal interval.-   114 Temporal sampling period.-   116 Light field frame.-   118 Spatial field.-   120 Spatial sampling period.-   122 Light field view image.-   124 Angular field.-   126 Angular sampling period.-   128 Spectral radiance.-   130 Spectral interval.-   132 Spectral sampling basis.-   134 Radiance sample.-   136 Depth.-   138 Sampling focus.-   150 Light sensor array.-   152 Light sensor.-   154 Angular sampling beam.-   156 Angular sampling filter pinhole.-   158 Image plane.-   160 Spatial sampling filter lens.-   162 Spatial sampling beam.-   164 Image point.-   166 4D sampling beam.-   168 Object point.-   170 Object plane.-   180 Light emitter array.-   182 Light emitter.-   184 Angular reconstruction beam.-   186 Angular reconstruction filter pinhole.-   188 Spatial reconstruction filter lens.-   190 Spatial reconstruction beam.-   192 4D reconstruction beam.-   200 Light field display.-   202 Display output beam.-   204 Virtual point source.-   206 Wavefront.-   210 Light field display element.-   212 Element output beam.-   220 Light field camera.-   222 Camera input beam.-   224 Real point source.-   230 Light field camera element.-   232 Element input beam.-   240 Viewer eye.-   242 Eye object point.-   244 Eye pupil.-   246 Axial input beam.-   248 Eye image point.-   250 Viewer.-   252 Scene object.-   254 Display element focus.-   256 Viewer fixation point.-   258 Viewer eye object plane.-   300 Two-way light field display.-   310 Two-way light field display element.-   320 Network.-   322 Two-way display controller.-   324 Remote viewer.-   326 Virtual image of remote viewer.-   328 Local viewer.-   330 Virtual image of local viewer.-   332 Remote object.-   334 Virtual image of remote object.-   336 Local object.-   338 Virtual image of local object.-   340 Camera controller.-   342 Display controller.-   344 Tracking camera.-   400 First positive lens.-   402 Electrode.-   404 Convex part of variable negative lens.-   406 Variable negative lens.-   408 Electrode.-   410 Linear polarizer.-   412 Second positive lens.-   414 Output/input beam.-   416 Second variable negative lens.-   418 Switchable polarization rotator.-   500 Scanned output beam.-   502 Output view image.-   504 Line scanner.-   506 Frame scanner.-   508 2D scanner.-   510 Timing generator.-   512 External frame sync.-   514 Frame sync.-   516 Line sync.-   518 Sampling clock.-   520 Radiance controller.-   522 Beam generator.-   524 Radiance modulator.-   526 Output focus.-   528 Output focus controller.-   530 Output focus modulator.-   540 Color beam generator.-   542 Red beam generator.-   544 Red radiance modulator.-   546 Green beam generator.-   548 Green radiance modulator.-   550 Blue beam generator.-   552 Blue radiance modulator.-   554 First beam combiner.-   556 Second beam combiner.-   600 Scanned input beam.-   602 Input view image.-   604 Radiance sensor.-   606 Radiance sampler.-   608 Input focus.-   610 Input focus controller.-   612 Input focus modulator.-   614 Beamsplitter.-   700 Laser.-   702 Angular reconstruction filter.-   704 Variable output focus.-   706 Beamsplitter.-   708 Mirror.-   710 Biaxial scanning mirror.-   712 Mirror.-   714 Variable input focus.-   716 Fixed input focus.-   718 Aperture.-   720 Photodetector.-   730 Angular reconstruction filter lenslet.-   732 Collimated output beam.-   734 Angular reconstruction beamlet.-   740 Biaxial scanner platform.-   742 Biaxial scanner platform hinge.-   744 Biaxial scanner inner frame.-   746 Biaxial scanner inner frame hinge.-   748 Biaxial scanner outer frame.-   750 Biaxial scanner mirror support post.-   752 Biaxial scanner mirror.-   760 Stationary input beam.-   762 Shift-and-accumulate photodetector linear array.-   764 Photodetector linear array.-   766 Photodetector.-   768 Analog shift register.-   770 Analog shift register stage.-   772 Analog-to-digital converter (ADC).-   774 Beam energy sample value.-   800 Oscillating display panel.-   802 Oscillating display chassis.-   804 Oscillating display frame.-   806 Oscillating display cover glass.-   808 Support spring.-   810 Spring support bracket on panel.-   812 Spring support bracket on chassis.-   814 Actuator.-   816 Rod.-   818 Actuator support bracket on panel.-   820 Actuator support bracket on chassis.-   900 Detect face & eyes.-   902 Estimate gaze direction.-   904 Transmit eye positions & gaze direction.-   906 Autofocus in gaze direction.-   908 Estimate fixation point.-   910 Transmit eye positions & fixation point.-   912 Focus on fixation plane.-   920 Capture light field frame.-   922 Transmit light field frame.-   924 Resample light field frame.-   926 Display light field frame.-   930 Eye positions (datastore).-   932 Fixation point (datastore).-   934 Light field video (datastore).-   936 Resample light field frame with focus.-   938 3D animation model.-   940 Render light field frame with focus.-   950 Two-way panel controller.-   952 Two-way element controller.-   954 View image datastore.-   956 Two-way element controller block.-   958 2D image datastore.-   960 Collimated view image datastore.-   962 Network interface.-   964 Input video interface.-   966 Output video interface.-   968 Display timing generator.-   970 Panel motion controller.-   972 High-speed data bus.-   980 Display element field.-   982 Display element eye field.-   984 Foveal field.-   986 Partial view image datastore.-   988 Partial foveal view image datastore.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS Light FieldParameterization

FIG. 1A shows a representative ray 100 of a continuous 6D light field,traversing the boundary 102 of the volume of interest at an intersectionpoint 104. The radiance (L) of the ray 100 is a function of time (t),boundary position (via coordinates x and y), ray direction (via angles aand b), and wavelength (w).

While the radiance of the ray is strictly only defined at the boundary,i.e. at the intersection point 104, additional knowledge of thetransparency of the two volumes separated by the boundary can allow theray's radiance to be extrapolated in either direction.

Radiance is a measure of radiant power per unit solid angle per unitarea (measured in watts per steradian per square meter, W/sr/m̂2). For aninfinitesimal ray of a continuous light field, the radiance is definedfor an infinitesimal solid angle and area.

For eventual display to a human, the radiance is usually sampledsparsely using either a triplet of basis functions related to thetristimulus color response of the human visual system, or a single basisfunction related to the human luminance response. These basis functionsensure proper band-limiting in the wavelength (w) dimension. Forconvenience the wavelength dimension is usually left implicit in mostanalysis. Thus a 6D light field becomes a 5D light field.

The time dimension (t) may be sampled at discrete time steps to producea sequence of 4D light field frames analogous to 2D image frames in aconventional video sequence. To avoid motion blur, or just as a matterof practicality, proper band-limiting is often not applied to the timedimension when sampling or generating video, and this can lead toaliasing. This is typically ameliorated by sampling at a sufficientlyhigh rate.

References in the literature to a 4D light field (and in the presentspecification, where appropriate) refer to a 4D light field frame, i.e.defined at a particular instant in time, with an implicit wavelengthdimension.

FIG. 1B shows a class diagram for a sampled, i.e. discrete, 6D lightfield, structured as a light field video 110.

The light field video 110 consists of a sequence of light field frames116, ordered by time (t), and captured over a particular temporalinterval 112 with a particular temporal sampling period 114.

Each light field frame 112 consists of an array of light field viewimages 122, ordered by ray position (x and y), and captured over aparticular spatial field 118 with a particular spatial sampling period120.

Each light field view image 122 consists of an array of spectralradiances 128, ordered by ray direction (a and b), and captured over aparticular angular field 124 with a particular angular sampling period126.

Each spectral radiance 128 consists of a sequence of radiance (L)samples 134, ordered by wavelength (w), and captured over a particularspectral interval 130 according to a particular spectral sampling basis132. The spectral radiance 128 has an optional depth 136, i.e. the depthof the scene in the ray direction, if known. The spectral radiance 128also records the sampling focus 138 with which it was captured. Thedepth 136 and sampling focus 138 are discussed further below.

Each radiance (L) sample 134 records a scalar radiance value.

In this specification the term “beam” is used to refer to a bundle ofrays, whose characteristics vary but are qualified in each context.

Light Field Sampling

FIGS. 2A, 2B, 3A and 3B illustrate an approach to band-limiting andsampling a continuous light field to obtain a discrete light field.

FIG. 2A shows a light sensor array 150 sampling the continuous lightfield with respect to ray direction for a particular ray position 104.Each light sensor 152 of the array 150 samples a particular raydirection, and integrates the beam 154 surrounding the nominal ray 100.This integration effects 2D low-pass filtering with respect to raydirection. The effective filter kernel is a non-ideal box filtercorresponding to the spatial extent of the light sensor 152. The lightsensors are ideally closely packed to ensure adequate filter support.The angular sampling beam 154 is focused at an infinitesimal pinholeaperture 156, which coincides with the ray position 104 on the boundary102.

The light sensor array 150 lies in a plane 158, parameterized by raydirection angles a and b.

The angular field 124 is the angle subtended at the angular samplingfilter pinhole 156 by the light sensor array 150. The angular samplingperiod 126, i.e. the inverse of the angular sampling rate, is the anglesubtended by the center-to-center spacing of the light sensors 152. Theangular sample size (i.e. the filter support) is the angle subtended bythe extent of the light sensor 152. The angular sample count equals theangular field 124 divided by the angular sampling period 126, i.e. thenumber of light sensors 152.

FIG. 2B shows an array of lenses sampling the continuous light fieldwith respect to ray position at the boundary 102. Each lens 160 of thearray samples a particular ray position, and integrates the parallelbeam 162 surrounding the nominal ray 100 by focusing the beam to a point164 on the light sensor 152. This integration effects 2D low-passfiltering with respect to position. The effective filter kernel is anon-ideal box filter corresponding to the spatial extent of the apertureof the spatial sampling filter lens 160. The lenses are ideally closelypacked to ensure adequate filter support.

The image distance is the distance from the second principal point ofthe lens 160 to the image plane 158.

The spatial field 118 equals the extent of the bounding surface 102. Thespatial sampling period 120, i.e. the inverse of the spatial samplingrate, is the center-to-center spacing of the spatial sampling filterlenses 160. The spatial sample size (i.e. the filter support) is thearea of the aperture of the lens 160. The spatial sample count equalsthe spatial field 118 divided by the spatial sampling period 120, i.e.the number of lenses 160.

FIG. 3A shows the combined effect of the spatial extent of the lightsensor 152 and the aperture of the lens 160 integrating sampling beam166 to effect 4D low-pass filtering, i.e. with respect to direction andposition simultaneously. The effective filter kernel is a 4D box filter,which provides reasonable but non-ideal band-limiting. It is difficultto do better than a box filter when integrating light spatially.

The scalar value obtained from the light sensor 152 is typicallyproportional to the time-integral of radiant power, i.e. radiant energy.It is convertible to a radiance sample 134 by dividing it by the 5Dsample size (i.e. 1D exposure duration, 2D spatial sample size, and 2Dangular sample size).

Note that the size of the light sensor 152 in the figures is exaggeratedfor clarity, and that the divergence of the (otherwise parallel) beam166 due to angular sampling is therefore also exaggerated.

Low-pass filtering of a light field results in visible blurring. In thepresent sampling regime, blur is proportional to the diameter of beam166. This has two additive components: the angular sampling blur, whichcorresponds to the angular sampling filter, i.e. the diameter of angularsampling beam 154 in FIG. 2A; and the spatial sampling blur, whichcorresponds to the spatial sampling filter, i.e. the diameter of spatialsampling beam 162 in FIG. 2B.

FIG. 3B shows beam 166 focused at a point 168 in object space using alens 160 with higher power than the lens 160 in FIG. 3A. Thecorresponding object distance is the distance from the object point 168to the first principal point of the lens 160. At the object point 168(and in general on the object plane 170) the spatial sampling blur iszero, and the beam diameter corresponds to the angular sampling bluralone.

The object sampling period, i.e. at the object plane 170, equals the(tangent of the) angular sampling period 126 multiplied by the objectdistance.

When the object plane 170 is at infinity then the sampling beam 166 ofFIG. 3A is obtained.

The convergence angle of the sampling beam 166 (or more properly thespatial sampling beam 162) is the angle subtended by the aperture of thelens 160 at the object point 168. Depth of field refers to a depthinterval, bounded by a given threshold spatial sampling blur (or defocusblur), bracketing the object point 168. The larger the convergence anglethe more rapidly defocus blur changes with depth, and hence theshallower the depth of field (i.e. the shorter the interval). Depth offield is relatively shallower for object distances that are shorter andfor apertures that are larger (i.e. corresponding to lower spatialsampling rates).

Adjusting the focus of the sampling beam 166 allows defocus blur at onedepth to be eliminated at the expense of increasing defocus blur atother depths, while maintaining proper support for the 4D low-passfilter. This allows defocus blur to be traded between different regionsof the light field, which is useful when blur minimisation is moreimportant in some regions than others (e.g. regions corresponding to thesurfaces of objects).

Changing focus does not affect the field of view or the total capturedradiance, since each lens 160 captures essentially the same set of raysindependent of focus.

If the sampling beam 166 is focused at infinity (as shown in FIG. 3A)its spatial sampling blur is constant and corresponds to the aperture ofthe lens 160. Since angular sampling blur increases with objectdistance, the relative contribution of this constant spatial samplingblur decreases with distance. This indicates that there is a thresholdobject distance beyond which angular sampling blur becomes dominant, andthat minimising blur by focusing the sampling beam 166 providesdiminishing returns as the object distance increases beyond thisthreshold distance.

The focus of beam 166 is recorded in the discrete light field 110 as thesampling focus 138 associated with the spectral radiance 128.

The optional depth 136 may be determined by range-finding (discussedbelow), and the sampling focus 138 may correspond to the depth 136, e.g.when beam 166 is focused according to scene depth.

In the well-known two-plane parameterization of the 4D light field[Levoy96], the uv plane coincides with the light field boundary 102 andthe st plane coincides with the object plane 170 (or equivalently theimage plane 158). The st plane is typically fixed, corresponding tofixed-focus sampling.

Light Field Reconstruction

The sampling regime used to capture a discrete light field 110,including the focus 138 of each sample, is used as the basis forreconstructing the corresponding continuous light field.

A continuous physical 4D light field is reconstructed from a discrete 4Dlight field using a 4D low-pass filter. The filter ensures that thecontinuous light field is band-limited to the frequency content of theband-limited continuous light field from which the discrete light fieldwas sampled.

FIGS. 4A, 4B, 5A and 5B illustrate an approach to band-limiting andreconstructing a continuous light field from a discrete light field.These figures mirror FIGS. 2A, 2B, 3A and 3B respectively, and the samereference numerals are used for corresponding parts where appropriate.

FIG. 4A shows a light emitter array 180 reconstructing a continuouslight field with respect to ray direction for a particular ray position104. Each light emitter 182 of the array 180 reconstructs a particularray direction, and generates the beam 184 surrounding the nominal ray100. This generation effects 2D low-pass filtering with respect to raydirection. The effective filter kernel is a non-ideal box filtercorresponding to the spatial extent of the light emitter 182. The lightemitters are ideally closely packed to ensure adequate filter support.The angular reconstruction beam 184 is focused at an infinitesimalpinhole aperture 186, which coincides with the ray position 104 on theboundary 102.

FIG. 4B shows an array of lenses reconstructing the continuous lightfield with respect to ray position at the boundary 102. Each lens 188 ofthe array reconstructs a particular ray position, and generates theparallel beam 190 surrounding the nominal ray 100 by focusing from point164 on the light emitter 182. This generation effects 2D low-passfiltering with respect to position. The effective filter kernel is anon-ideal box filter corresponding to the spatial extent of the apertureof the lens 188. The lenses are ideally closely packed to ensureadequate filter support.

FIG. 5A shows the combined effect of the spatial extent of the lightemitter 182 and the aperture of the lens 188 generating reconstructionbeam 192 to effect 4D low-pass filtering, i.e. with respect to directionand position simultaneously. The effective filter kernel is a 4D boxfilter, which provides reasonable but non-ideal band-limiting. It isdifficult to do better than a box filter when generating lightspatially.

The scalar value provided to the light emitter 182 is typicallyproportional to emitter power. The radiance sample 134 is convertible toemitter power by multiplying it by the 5D sampling period (i.e. the 1Dtemporal sampling period 114, the 2D spatial sampling period 120, andthe 2D angular sampling period 126), and dividing it by the actualon-time of the emitter (which is typically shorter than the temporalsampling period 114). Note that if the 4D (spatial and angular)reconstruction filter support is smaller than the 4D sampling periodthen the same radiant power is simply delivered via a more compact beam.

Proper 4D reconstruction relies on the light emitter 182 emitting allpossible rays between the extent of the light emitter 182 and theaperture of the lens 188. This is satisfied if the emitter 182 isdiffuse.

FIG. 5B shows beam 192 focused from a virtual object point (to the leftof the array 180, and not shown in FIG. 5B, but coinciding with objectpoint 168 in FIG. 6B) using a lens 188 with lower power than the lens188 in FIG. 5A.

When the virtual object plane is at infinity then the beam 192 of FIG.5A is obtained.

The divergence angle of the reconstruction beam 192 (or more properlythe spatial reconstruction beam 190) is the angle subtended by theaperture of the lens 188 at the virtual object point. The reconstructionbeam 192 has a depth of field, determined by its divergence angle,corresponding to the depth of field of the sampling beam 166 in FIG. 3B.

Adjusting the focus of reconstruction beam 192, per the sampling focus138, allows it to be matched to the sampling beam 166 used to create thesample value.

The reconstruction beams 192 of FIGS. 5A and 5B match the sampling beams166 of FIGS. 3A and 3B respectively, and this is illustrated explicitlyin FIGS. 6A and 6B, where the left side of each figure shows thesampling beam 166 and the right side shows the matching reconstructionbeam 192.

Light Field Display

FIG. 7A shows an idealized light field display 200 emitting output beams202 corresponding to two virtual point sources 204 constituting a verysimple virtual scene. Each output beam 202 consists of sphericalwavefronts 206, each with its origin at respective point source 204. Theexit pupil of each output beam 202 at the surface of the display 200equals the extent of the entire display.

For clarity, FIG. 7A shows only two point sources 204. In practice thedisplay 200 would emit beams from a continuous set of point sources.Also, while not explicitly shown, the radiance cross-section of eachbeam 202 could be non-uniform.

To an observer situated in front of the light field display 200, thedisplay 200 would appear indistinguishable from a window onto a realscene containing the point sources 204.

While FIG. 7A shows display 200 emitting diverging beams correspondingto virtual point sources 204 located behind the display, the display 200could also emit converging beams corresponding to virtual point sourceslocated in front of the display.

FIG. 7B shows a realization of the display 200, segmented into an arrayof contiguous display elements 210, each of which performs thereconstruction functions of the light emitter array 180 and lens 188 inFIG. 5B.

Each display element 210 is shown emitting output beams 212corresponding to the point sources 204, i.e. each display element 210behaves in the same way as the overall display 200, but with a reducedexit pupil equal to the extent of the display element 210.

Each output beam 212 emitted by a display element 210 in FIG. 7B isfocused at its respective point source 204, thus the output beams 212abut to form the wider output beams 202 emitted by the entire display200 in FIG. 7A, with the same wavefronts 206.

The segmented light field display 200 is configured to directly displaya discrete 6D light field 110. During display, the surface of thedisplay 200 corresponds to the light field boundary 102 associated withthe discrete light field, and the position of each display element 210corresponds to a sampling position 104 (x, y) on the boundary. Thedirection of each beam 212 emitted by the display element corresponds toa sampling direction (a, b), and the average radiance of each beam 212corresponds to the sampled spectral radiance 128. The focus of each beam212 corresponds to the sampling focus 138.

Thus each display element 210 reconstructs, at a given time, thecontinuous light field corresponding to a single light field view image122, and the entire display 200 reconstructs, at a given time, thecontinuous light field corresponding to a single light field frame 116.The display 200 thus reconstructs, over time, the continuous 6D opticallight field corresponding to the discrete 6D light field video 110.

For clarity, the spatial sampling period 120 illustrated in FIG. 7B isrelatively large, while the angular sampling period 126 is relativelysmall. Thus the output beams 212, each of which is associated with asingle spectral radiance 128 within the discrete light field 110, areshown to converge exactly at their respective virtual point source 204.In practice the beams converge in a finite area rather than at a point,i.e. the point source is blurred in proportion to the angular samplingperiod 126.

As is evident from FIG. 7B, the larger the spatial sampling period 120the less angular object detail is displayed, and the larger the angularsampling period 126 the less spatial object detail is displayed. Theformer manifests as shallow depth of field, while the latter manifestsas blur in the object plane.

The smaller the 4D sampling period (i.e. the higher the 4D samplingrate) the greater the fidelity of the light field display. However, fora fixed number of samples, it is possible to reduce object-plane blur atthe cost of shallower depth of field.

Light Field Camera

FIG. 8A shows an idealized light field camera 220 capturing input beams222 corresponding to two real point sources 224 constituting a verysimple real scene. Each input beam 222 consists of spherical wavefronts,each with its origin at respective point source 224. The entry pupil ofeach input beam 222 at the surface of the camera 220 equals the extentof the entire camera.

For clarity, FIG. 8A shows only two point sources 224. In practice thecamera 220 would capture beams from a continuous set of point sources.Also, while not explicitly shown, the radiance cross-section of eachbeam 222 could be non-uniform.

FIG. 8B shows a realization of the camera 220, segmented into an arrayof contiguous camera elements 230, each of which performs the samplingfunctions of the light sensor array 150 and lens 160 in FIG. 3B.

Each camera element 230 is shown capturing input beams 232 correspondingto the point sources 224, i.e. each camera element 230 behaves in thesame way as the overall camera 220, but with a reduced entry pupil equalto the extent of the camera element 230.

Each input beam 232 captured by a camera element 230 in FIG. 8B isfocused at its respective point source 224, thus the input beams 232abut to form the wider input beams 222 captured by the entire camera 220in FIG. 8A, with the same wavefronts.

The segmented light field camera 220 is configured to directly capture adiscrete 6D light field 110. During capture, the surface of the camera220 corresponds to the light field boundary 102 associated with thediscrete light field, and the position of each camera element 230corresponds to a sampling position 104 (x, y) on the boundary. Thedirection of each beam 232 captured by the display element correspondsto a sampling direction (a, b), and the average radiance of each beam232 is captured as the spectral radiance 128. The focus of each beam 232corresponds to the sampling focus 138.

Thus each camera element 230 samples, at a given time, the continuouslight field corresponding to a single light field view image 122, andthe entire camera 220 samples, at a given time, the continuous lightfield corresponding to a single light field frame 116. The camera 220thus samples, over time, the continuous 6D optical light fieldcorresponding to the discrete 6D light field video 110.

For clarity, the spatial sampling period 120 illustrated in FIG. 8B isrelatively large, while the angular sampling period 126 is relativelysmall. Thus the input beams 232, each of which is associated with asingle spectral radiance 128 within the discrete light field 110, areshown to converge exactly at their respective real point source 224. Inpractice the beams converge in a finite area rather than at a point,i.e. the point source is blurred in proportion to the angular samplingperiod 126.

Non-Planar Light Field Boundary

Although the figures show the light field boundary 102 associated withthe light field display 200 and the light field camera 220 as planar, itmay in practice assume any convenient shape.

Depth Perception

Creatures with foveal vision (such as humans) fixate on a point byrotating the eye (or eyes) so that the image of the point is centered onthe high-density foveal region of the retina. This maximises thesharpness of the perceived image. When the retinal images of two eyesare mentally fused into a single image during the process of stereopsis,the degree of eye convergence (or vergence) provides a crucial cue tothe absolute depth of the fixation point.

In addition to rotating the eye(s) during fixation, creatures alsoadjust the shape of the lens of the eye to bring the point of fixationinto focus on the retina. In this process of accommodation, the state ofthe muscles controlling the lens provides another important cue toabsolute depth.

The human accommodation response curve shows over-accommodation to farstimuli and under-accommodation to near stimuli, with a typicalcross-over (i.e. perfect accommodation) at an object distance of around50 cm, and a typical minimum response of 0.5 diopters (2 m) for objectdistances greater than 2-3 m [Ong93, Palmer99, Plainis05]. Crucially,then, the human visual system never accommodates properly to farstimuli.

The vergence and accommodation responses are closely coupled, and anymismatch between the vergence and accommodation cues provided by adisplay can lead to viewer discomfort [Hoffman08].

Parallax refers to the difference in apparent position of an object whenviewed from different viewpoints, with close objects exhibiting greaterparallax than distant objects. Binocular disparity due to parallaxsupports relative depth perception during stereopsis, i.e. relative tothe absolute depth of fixation. Motion parallax supports relative depthperception even with one eye.

Perception of a Focused Light Field

As illustrated in FIG. 7B, each output beam 212 corresponding to a pointsource 204 has its origin at the point source, i.e. each constituent rayof the beam 212 originates at the point source 204. Equivalently, thespherical wavefronts 206 of the beam 212 have their center of curvatureat the point source 204. This ensures that a viewer perceives theparallax of point source 204 correctly both within any given beam 212and across multiple beams 212, resulting in accurate binocular disparityand smooth motion parallax. The smaller the object distance the greaterthe divergence of each beam 212, and hence the more important thepresence of intra-beam parallax. By contrast, fixed-focus 3D displaysonly provide parallax between different views, and provide incorrect(and therefore conflicting) parallax within any given view. Furthermore,autostereoscopic displays typically provide a modest number of views,resulting in only approximate binocular parallax and discontinuousmotion parallax.

The correctly-centered spherical wavefronts 206 of the beams 212 alsoallow the viewer to accommodate to the correct depth of thecorresponding point source 204, ensuring that the viewer's vergence andaccommodation responses are consistent. This avoids thevergence-accommodation conflicts associated with fixed-focus 3Ddisplays.

Using a relatively high angular sampling rate decouples the angularresolution of a light field display from the spatial sampling rate (seebelow). This contrasts with typical 3D displays where the spatialsampling rate determines the angular display resolution. For the presentdisplay 200, this allows the spatial sampling rate to be lower than withfixed-focus 3D displays. For a given overall (4D) sampling rate this inturn allows a relatively higher angular sampling rate.

The angular resolution of a focused light field display 200, whendisplaying a virtual object at a particular object distance (r) behindthe display, and viewed at a particular distance (d) in front of thedisplay, is the angle (g) subtended, at the viewpoint, by one objectsampling period (h) (i.e. on the object plane), i.e. g=h/(r+d) (forsmall g).

The object sampling period (h) is a function of the angular samplingperiod 126 (q) and the object distance (r), i.e. h=qr (for small q).Hence g=qr/(r+d).

The angular sampling period 126 (q) therefore represents the minimumlight field display resolution. As the object distance (r) approachesinfinity or the viewing distance (d) approaches zero (i.e. in both casesas r/(r+d) approaches one) the display resolution converges with theangular sampling period 126 (q).

The light field display 200 can therefore be configured to match thehuman perceptual limit, for any viewing geometry, by configuring itsangular sampling period 126 (q) to match the maximum angular resolutionof the eye (about 60 cycles per degree [Hartridge22], equivalent to anangular sampling period of approximately 0.008 degrees). For a 40-degreefield of view this equates to an angular sample count of 4800.

The light field display resolution for a given viewing distance (d) andobject distance (r) can significantly exceed the angular sampling period126 (q) when the viewing distance exceeds the object distance. Forexample, if the viewing distance is four times the object distance, thedisplay resolution is five times the angular sampling period 126, andfor a 40-degree angular field 124 an angular sample count of 960 issufficient to match the human perceptual limit.

If the angular sampling period 126 (q) is sufficiently large (such asfor typical autostereoscopic displays) then the spatial sampling period120 (s) determines the angular display resolution (g). The angularresolution (g) is then the angle subtended by one spatial samplingperiod 120 (s) at the display surface, i.e. g=s/d (for small g). Thecomplete equation for the angular resolution of a light field display isthen: g=min(s/d, qr/(r+d)).

The foregoing calculations represent the best case, in that they ignorethe imperfect human accommodation response. The perceived resolution ofa light field display can be improved by (at least partially) matchingits focus to the actual human accommodation response to a given depthstimulus, rather than to the depth itself. This can include matching theknown accommodation response of an individual viewer (including theeffect of spectacles, if worn). However, any deviation in focus from theproper depth-determined focus leads to parallax error, and this errorincreases with decreasing object distance. With increasing objectdistance, however, parallax error is increasingly masked by angularsampling blur. A compromise, then, is to select a threshold objectdistance beyond which light field focus is fixed. This divides the lightfield focus regime into a fixed-focus far-field regime and avariable-focus near-field regime. The fixed-focus far-field thresholdcan be as close as the typical minimum accommodation response (2 m), orsignificantly larger (including, in the limit, infinity).

Equivalence of Scene Focus and Viewer Focus

FIG. 9A shows the eye 240 of a viewer located in the reconstructed lightfield of a virtual point source 204. The light field is reconstructed bysegmented display 200. The eye is focused at an object point 242coinciding with the virtual point source 204. The input beam 246admitted by the pupil of the eye, a sub-beam of one of the output beams212, is focused to a point 248 on the retina. The image of the pointsource 204 on the retina is therefore sharp.

FIG. 9B shows the object point 242 now closer to the display 200 thanthe virtual point source 204. The image point 248 corresponding to thepoint source 204 is now in front of the retina, and the image of thepoint source on the retina is therefore blurred. This is as it shouldbe, i.e. it matches reality.

FIG. 9C shows the display 200 now displaying the light field of a pointsource coinciding with the translated object point 242. The input beam246 is now focused at object point 242 rather than original point source204, so is once again in focus on the retina (at image point 248). Sincethe input beam is not in focus at point source 204, the image of pointsource 204 on the retina remains blurred (and by the same amount as inFIG. 9B). This is again as it should be.

For clarity, FIGS. 9A through 9C only show a single object point 242, onthe optical axis of the eye 240. The “plane” of focus is the locus ofall such points, and is an approximately spherical surface with a radiusequal to the object distance, centred at the first nodal point of theeye.

The equivalence of what the viewer perceives in FIGS. 9B and 9Cindicates that there are two useful modes of operation for displaying afocused light field. In the first mode the display is focused on objectsin the scene. In the second mode the display is focused according to theviewer's focus.

Light Field Display Focus Strategies

The advantage of scene-based focus is that the reconstructed light fieldis intrinsically multi-viewer. One disadvantage is that the depth of thescene must be known or determined (discussed below). Anotherdisadvantage is that output focus may need to be varied for each sample,requiring fast focus switching. In addition, a single depth needs to bechosen for each sample, and this may require a compromise whensignificant depth variations are present within the sampling beam.

If the focus modulation rate of the display element 210 is significantlylower than the sampling rate 114, then multiple depths can be supportedvia multiple display passes, i.e. one pass per depth. The output focusof each display element 210 is then adjusted for each pass according toits corresponding scene depth in that pass. However, because the numberof distinct depths within a view image 122 is typically larger than thepractical number of display passes, the set of depths supported for agiven display element is likely to be a compromise. One way to choosethe set of depths is to estimate the full range of depths within theview image 122 of a display element and then identify the most commondepth clusters. Intermediate depths can then be displayed usingdepth-weighted blending [Hoffman08].

The advantage of viewer-specific focus is that focus can be variedrelatively slowly, and depth variations within a single sample areintrinsically correctly handled. The disadvantage is that thereconstructed light field is viewer-specific, and that the viewer musttherefore be tracked. It has the additional disadvantage that the lightfield must be captured (or synthesized) with the correct focus, orrefocused before display.

The sharpness of the refocused light field can be increased by recordingmultiple spectral radiance samples 128 per direction (a, b), each with adifferent sampling focus 138. Sharpness is particularly increased ifeach sampling focus 138 corresponds to an actual object depth within thesampling beam 166, whether directly or via a transmitted or reflectedpath.

The viewer-specific light field view image 122 for each display element210 is obtained by integrating, for each direction, all rays passingthrough the object point 242 (or disc, more properly) for that directionand through the aperture of the display element. When the light field110 is captured via a light field camera 220, this integration may beperformed by focusing each camera element 230 accordingly.

In the viewer-specific focus mode, then, the fixation point of theviewer is constantly tracked, and each display element 110 isindividually controlled to emit a viewer-specific light field focusedaccording to the depth of the fixation point.

Multiple viewers can be supported via multiple display passes, i.e. onepass per viewer. Alternatively, display focus can be controlled by asingle user, and other users can passively view the display at thatfocus, i.e. in the same way they would view a fixed-focus light fielddisplay.

In a hybrid mode, one or more display passes may be viewer-specific,while one or more additional display passes may be scene-based. Forexample, two display passes can be used to provide a viewer-specificpass, a finite-focus pass for near scene content, and an infinite-focuspass for far scene content.

During an optimised viewer-specific display pass output is onlygenerated in the direction of the viewer, as discussed further below inrelation to FIG. 42A. This means that a viewer-specific display pass isonly visible to the target viewer, and may only consume a fraction ofthe frame period, depending on the implementation of the display element210.

A viewer-specific display pass will typically utilise less than 10% ofthe angular field 124, and if the display element 210 is scanning (asdescribed in detail further below), then, at least in one dimension, thedisplay pass will only consume a corresponding fraction of the frameperiod. A reduced-duration viewer-specific frame is referred to as asub-frame hereafter.

Unlike traditional head-tracking 3D displays where the displayed contentis viewer-specific, a light field display 200 operating inviewer-specific mode displays viewer-independent content withviewer-specific focus. If the viewer changes their point of fixation ormoves relative to the display then the display focus may need to beupdated, but this can happen relatively slowly because the viewer isalways embedded in a valid (if not necessarily completely optimal)reconstructed light field, and the human accommodation response isrelatively slow (i.e. of the order of several hundred milliseconds).

Viewer-Specific Focus Modes

FIGS. 10A through 10D illustrate two strategies for displaying aviewer-specific light field.

FIG. 10A shows a viewer 250 gazing at a light field display 200 emittinga light field corresponding to a virtual scene consisting of severalobjects 252. A tracking system incorporated in or associated with thedisplay 200 tracks the face of the viewer 250 and hence the locations ofthe viewer's two eyes 240.

FIG. 10B shows the location of one of the eyes 240 used to determine aviewing direction through each display element 210, and thus, for eachviewing direction, an intersection point 254 with a scene object 252.The focus of each display element is shown set according to the depth ofthe corresponding intersection point 254.

FIG. 10C shows the tracking system used to track the gaze direction ofeach of the viewer's two eyes 240, and hence to estimate their fixationpoint 256. Assuming fixation and accommodation are synchronised, as theyare under normal circumstances, the viewer's focus can be estimated fromthe depth of the fixation point 256.

FIG. 10D shows the plane of focus 258 of one of the eyes 240, estimatedfrom the depth of the fixation point 256, and, for each viewingdirection, an intersection point 254 with the plane of focus. The focusof each display element is again shown set according to the depth of thecorresponding intersection point 254.

The first viewer-specific mode, shown in FIG. 10B, represents a hybridmode which relies on scene depth information and face detection, butdoes not require gaze estimation. It is referred to as theposition-based viewer-specific focus mode.

The second viewer-specific mode, shown in FIGS. 10C and 10D, does notrely on scene depth information but does require gaze estimation. It isreferred to as the gaze-directed viewer-specific focus mode.

Although FIG. 10D shows the output focus set according to the positionof an individual eye 240, for fixation depths that are large comparedwith the distance separating the eyes the output focus of a particulardisplay element 210 will differ sufficiently little between the two eyesthat an average output focus can be used to serve both eyes during asingle display pass. Any display element 210 that contributes to fovealvision in one or the other eye (as discussed later in this specificationin relation to FIG. 42B) should, however, be focused for thecorresponding eye.

The position-based and gaze-directed focus modes are complementary. Thegaze-directed mode produces more accurate focus, but relies on gazeestimation which becomes decreasingly tractable as the distance betweenthe viewer and the display increases. The position-based mode relies onface detection, which remains tractable over larger distances, and theaccuracy of position-based scene focus increases with distance, sincethe angle subtended by a display element 210 decreases with distance.

The two modes can therefore be used in tandem, with the operative modeselected individually for each viewer according to the distance betweenthe display and the viewer.

Choice of Focus Strategy

A suitable focus strategy depends on how the display is used, i.e. thenumber of viewers, their typical viewing distances, and the nature ofthe displayed scenes. It also depends on the capabilities of aparticular implementation of the light field display 200, in particularon the focus modulation rate.

The minimum viewing object distance is the sum of the minimum displayedobject distance and the minimum viewing distance. If the minimum viewingobject distance is larger than the far-field threshold then a singlefixed-focus display pass is sufficient.

If the minimum displayed object distance is larger than the far-fieldthreshold then the far-field regime applies independent of viewingdistance, and viewers need not be tracked. For example, the display 200may be simulating a window onto a distant exterior scene.

If the minimum displayed object distance is smaller than the far-fieldthreshold then the near-field regime applies wherever the minimumviewing object distance is smaller than the far-field threshold, andviewers may need to be tracked.

If the focus modulation rate of the light field display 200 matches thesampling rate 114 then a viewer-independent near-field light field canbe displayed in a single pass.

If the light field display 200 is used as a near-eye display (NED) thenthere is only a single viewing eye. The gaze-directed viewer-specificfocus mode may be effectively used, e.g. based on the fixation depthinferred from the vergence of the two eyes, and the focus modulationrate only has to match the relatively slow human accommodationmechanism, which takes several hundred milliseconds to refocus (lessthan 4 Hz).

If the light field display 200 is used by multiple relatively closeviewers, then multiple passes of gaze-directed viewer-specific focus canbe effectively utilised.

If the display 200 supports sub-frames then multiple display passes maybe made during a single frame duration. If not, then the number ofdisplay passes is limited by the ratio of the temporal sampling interval114 to the frame duration (assuming the temporal sampling interval 114is perceptually based and therefore cannot be compromised).

If the eye-specific sub-frame period is Teye, the focus switching timeis Tfocus, the frame period is Tframe, the number of full-frame passesis Nfull, and the temporal sampling period 114 is Ts, then the availablenumber of eye-specific passes Neye is given by:Neye=floor((Ts−(Tframe*Nfull))/(Tfocus+Teye))

For illustrative purposes it is assumed that the frame period Tframe ishalf the sampling period Ts. This allows two full-frame passes when thenumber of eye-specific passes Neye is zero, and the following number ofeye-specific passes when the number of full-frame passes Nfull is one:Neye=floor(Tframe/(Tfocus+Teye)). Hence Tfocus=(Tframe/Neye)−Teye.

For illustrative purposes it is further assumed that the required numberof eye-specific passes Teye is four, and that the sub-frame durationTeye is 10% of Tframe. The maximum allowed focus switching time Tfocusis then given by: Tfocus=Tframe*0.15.

Assuming a frame rate of 100 Hz, i.e. a frame period Tframe of 10 ms(corresponding to a temporal sampling period Ts 114 of 20 ms (50 Hz)),this equates to a focus switching time Tfocus of 1.5 ms. Assuming aframe rate of 200 Hz, it equates to a focus switching time Tfocus of 750us.

If the display element 210 is scanning, and it is assumed that viewersare distributed horizontally with respect to the display 200, then it isadvantageous to assign the fast scan direction to the vertical dimensionof the display to allow focus to be varied horizontally, i.e. in theslow scan direction, during a single display pass (assuming sufficientlyfast focus switching). This allows multiple eye-specific focus zones tobe created during a single (full-frame) display pass, and provides analternative to making multiple viewer-specific sub-frame display passes.

The choice of focus strategy during capture by a light field camera 220follows the same principles as discussed above in relation to display bya light field display 200. This includes adjusting the capture focusaccording to the position and/or gaze of one or more viewers of a lightfield display 200, i.e. if the camera 220 is capturing a light fieldthat is being displayed in real time by the light field display 200, asdiscussed in more detail below.

Depth Estimation

The optional depth 136 associated with the spectral radiance 128 recordsthe scene depth within the sampling beam 166. It may represent acompromise when significant depth variations are present within thesampling beam, e.g. due to partial occlusions, transparency orreflections. For example, it may represent the depth to the firstsufficiently opaque surface along the nominal sampling ray 100.Alternatively, as discussed above, multiple depths 136 may be recordedfor each direction (a, b).

The depth 136 may be used for a number of purposes, including displayingthe light field with scene-based focus (as discussed above), estimatingthe fixation point of a viewer (discussed below), light fieldcompression (discussed below), and depth-based processing andinteraction in general.

When the light field 110 is synthetic, i.e. generated from a 3D model,the depth of the scene is known. When the light field 110 is capturedfrom a real scene, the depth may be determined by range-finding.

Range-finding may be active, e.g. based on time-of-flight measurement[Kolb09, Oggier11], or passive, e.g. based on image disparity[Szeliski99, Seitz06, Lazaros08] or defocus blur [Watanabe96]. It mayalso be based on a combination of active and passive techniques[Kolb09]. Range-finding is discussed further below.

Two-Way Light Field Display

It is advantageous to combine the functions of a light field display 200and a light field camera 220 in a single device, due both to thesymmetry of application and the symmetry of operation of the twodevices. Such a device is hereafter referred to as a two-way light fielddisplay.

FIG. 11 shows a pair of two-way light field displays 300 connected via anetwork 320. Each two-way light field display 300 is segmented into anarray of contiguous two-way light field display elements 310, each ofwhich performs the functions of light field display element 210 andlight field camera element 220.

The figure shows a remote viewer 324, at the top, interacting with theremote two-way light field display 300, and a local viewer 328, at thebottom, interacting with the local two-way light field display 300. Eachtwo-way display 300 is controlled by a respective display controller322, described in more detail later in this specification.

The remote viewer 324 is accompanied by a remote object 332, while thelocal viewer 328 is accompanied by a local object 336. The local viewer328 is shown fixating on a virtual image 334 of the remote object 332,while the remote viewer 324 is shown fixating on a virtual image 338 ofthe local object 336. The remote display 300 also displays a virtualimage 330 of the local viewer, and the local display 300 displays avirtual image 326 of the remote viewer 324.

Each viewer may be tracked by the display controller 322 of theirrespective two-way display 300, using view images 122 captured via thetwo-way display 300 (or via separate tracking cameras, discussed below).As previously described (and described in more detailed further below),each viewer's face position or gaze direction may be used to control thecapture focus of the corresponding two-way light field display 300.

The use of a pair of two-way light field displays 300 rather thanconventional displays and cameras allows significantly improvedcommunication between the remote viewer 324 and local viewer 328,promoting a strong sense of shared presence. For example, each viewercan determine where the other viewer is looking or pointing, and objectscan be held up close to the surface of the two-way display 300 for closeinspection by the other viewer.

FIG. 11 also makes it clear that if the two two-way displays 300 aremounted back-to-back then they function as a virtual two-way window,i.e. they (and the intervening space) become effectively invisible.

FIG. 12 shows a one-way configuration, consisting of a remote lightfield camera 220, at the top, and a local light field display 200, atthe bottom, connected via a network 320.

The figure shows a local viewer 328, at the bottom, viewing the display200. The light field camera 220 is controlled by a camera controller340, while the light field display 200 is controlled by a displaycontroller 342. The controllers are described in more detail later inthis specification.

The remote scene contains a remote object 332, and the local viewer 328is shown fixating on a virtual image 334 of the remote object 332.

The viewer 328 may be tracked by the display controller 342, usingimages captured via two or more tracking cameras 344 connected to thecontroller 342. As previously described, the viewer's face position orgaze direction may be used to control the capture focus of the lightfield camera 220.

In the remainder of this specification any reference to a light fielddisplay 200 (and light field display element 210) should be taken asequivalent to the display function of a two-way light field display 300(and two-way light field display element 310), and vice versa. Likewise,any reference to a light field camera 220 (and light field cameraelement 230) should be taken as equivalent to the camera function of atwo-way light field display 300 (and two-way light field display element310), and vice versa.

Face Detection and Gaze Estimation

As discussed above, the light field display 200 may use knowledge of theposition and gaze direction of a viewer to generate a viewer-specificoutput light field, including with viewer-specific focus and aviewer-specific angular field.

Depending on the distance between a viewer and the light field display200, the display can variously make use of knowledge of thethree-dimensional position of the viewer, the positions of the viewer'seyes, the lines of gaze of the eyes, the fixation depth of the eyes, andthe fixation point of the eyes, to generate viewer-specific output. Theviewer's gaze direction may only be estimated with useful accuracy whenthe viewer is relatively close to the display, while the position of theviewer's face and eyes may be estimated with useful accuracy even whenthe viewer is relatively distant from the display.

Robust and high-speed face detection in digital images is typicallybased on a cascade of classifiers trained on a database of faces[Jones06]. Multiple face detectors can be trained and used together tocover a wide range of head poses [Jones03].

Approximate eye detection is typically intrinsic to face detection, andmore accurate eye positions can be estimated after face detection[Hansen10]. Detection is also easily extended to other useful featuresof the face and eyes, including the eyebrows, nose, mouth, eyelids,scleras, irises and pupils [Betke00, Lienhart03, Hansen10].

Face detection and subsequent feature detection is performed on imagesfrom multiple cameras to obtain estimates of feature positions in threedimensions, using images either from two or more calibrated trackingcameras 344, or from two or more light field camera elements 230 usedfor tracking (e.g. located at the corners of the light field camera220). The use of multiple tracking cameras also provides better coverageof potential viewer positions and poses. Feature positions may alsoestimated from depth data obtained by active range-finding (as discussedabove).

For the purposes of gaze estimation, the display 200 includes multiplenear-infrared (NIR) light sources to allow the line of gaze of each eyeto be estimated from the difference between the position of its pupiland the position of the specular reflection (glint) of each light sourceon its cornea [Shih00, Duchowski07, Hansen10]. The NIR light sources maybe powered only on alternate video frames to assist with the detectionof their reflections in an image [Amir03]. To assist with pupildetection the display 200 may incorporate an additional NIR lightsource, positioned on or close to the axis of one of the trackingcameras, to produce a bright retinal reflection through the pupil ofeach eye. This light source may be powered on alternate video frames tothe glint-producing light sources.

The line of gaze of an eye corresponds to the optical axis of the eye,while the desired line of sight is determined by the retinal position ofthe slightly off-axis fovea. The line of sight may be estimated from theline of gaze using an estimate of the position of the fovea. Theposition of the fovea can either be assumed (e.g. based on populationdata), or can be estimated via calibration. Explicit calibrationtypically requires the viewer to fixate on a set of targets. Implicitcalibration relies on inferring when the viewer is fixating on knownscene points. Calibration can be performed anew each viewing session, orcalibration data can be stored and retrieved when the viewer interactswith the display. For example, it may be retrieved based on recognisingthe viewer's face [Turk92, Hua11], or it may be based on another form ofidentification mechanism, such as a credential provided by the viewer.

The fixation point of the viewer may be estimated from the intersectionpoint of the lines of sight of the viewer's two eyes. The fixation pointmay be refined using knowledge of the depth of the scene, under theassumption that the viewer is likely to be fixating on a surface pointin the scene. Alternatively, the fixation depth may be estimated fromthe vergence of the two lines of sight, without estimating an explicitfixation point.

As an alternative to active gaze estimation using NIR illumination, gazeestimation may be passive, i.e. based only on images of the viewer'seyes under ambient illumination [Hansen10]. This relies on estimatingthe relative positions and shapes of key features such as the corners ofthe eyes, the eyelids, the boundary between the sclera and iris (thelimbus), and the pupil, relative to the overall pose of the head.Passive gaze estimation is generally less accurate than active gazeestimation.

For the purposes of both active and passive gaze estimation, the display200 may include additional steerable narrow-field-of-view (FOV) trackingcameras 344 for obtaining more detailed images of viewers' eyes.Selected camera elements 230, if scanning, may also be used as steerablenarrow-FOV tracking cameras by narrowing and angling their angularfields of view.

Two-Way Light Field Display Implementation

In a preferred embodiment the segmented two-way light field display 300captures and displays a light field video 110, i.e. a succession oflight field frames 116, and operates with a sufficiently short temporalsampling period 114 to minimise or eliminate perceived flicker, i.e.ideally at a frame rate of at least 60 Hz, the peak critical flickerfusion (CFF) frequency.

As a motivating example, and for the purposes of illustrativecalculations in the remainder of this specification, a two-way lightfield display 300 with the following parameters is used: a temporalsampling period 114 of 10 ms (i.e. a frame rate of 100 Hz, assuming oneframe per temporal sampling period); a spatial sampling period 120 of 2mm; a spatial field 118 (i.e. display surface extent) that is 1000 mmwide by 500 mm high; hence a spatial sample count of 500 by 250; anangular sampling period 126 of 0.04 degrees; an angular field of 40degrees by 40 degrees; hence an angular sample count of 1000 by 1000; anRGB spectral sampling basis 132; and 12-bit radiance 134 samples.

This illustrative two-way display 300 configuration has a throughput of4E13 radiance samples/s in each direction (i.e. display and capture).

Note that many applications allow significantly lower frame rates,sampling periods and sample counts.

Display Luminance and Power

The luminance of the daylight terrestrial sky ranges up to about 10,000cd/m̂2 (candela per square meter), which equates to a radiance (in thevisible spectrum) of about 15 W/sr/m̂2. Reproducing this using theillustrative display configuration equates to an output power of about20 uW (microwatts) per display element 310, and a total output power ofabout 3 W for the entire display 300. A typical indoor light source mayhave a luminance an order of magnitude larger, i.e. 100,000 cd/m̂2,equating to 200 uW per display element and 30 W for the entire display.

Any radiance samples 134 that exceed the maximum radiance of the display300 can be clamped, or all radiance values can be scaled within theavailable range.

Array-Based Two-Way Light Field Display Element

FIGS. 13A and 13B show a schematic diagram of one embodiment of atwo-way light field display element 310 of the two-way light fielddisplay 300.

The two-way element 310 consists of a light sensor array 150 overlaid bya transparent light emitter array 180. Focusing is provided by a firstfixed-focus positive lens 400, a variable-focus negative lens 406, and asecond fixed-focus positive lens 412.

The variable-focus negative lens 406 may be any suitable lens withcontrollable focus, as discussed in more detail in relation to thescanning light field display element later in this specification.

The variable-focus negative lens 406 shown in FIGS. 13A and 13B consistsof a nematic liquid crystal cell sandwiched between a concave face and aplanar face. The concave face is formed by an adjacent convex part 404.The liquid crystal is birefringent, with light polarized parallel to thedirector experiencing a higher (extraordinary) refractive index (n1e),and light polarized perpendicular to the director experiencing a lower(ordinary) refractive index (n1o). For illustrative purposes an ordinaryindex of 1.5 and an extraordinary index of 1.8 are used, parametersrepresentative of commercially-available liquid crystal materials suchas Merck E44.

The liquid crystal cell is further sandwiched between a pair oftransparent electrodes 402 and 408 (e.g. ITO). When no voltage isapplied across the electrodes, as illustrated in FIG. 13A, the director(indicated by the orientation of the small ellipses in the figure)follows the horizontal rubbing direction. When a saturation voltage isapplied, as illustrated in FIG. 13B, the director becomes verticallyaligned with the applied field.

The refractive index (n2) of the convex part 404 is approximatelymatched to the ordinary refractive index (n1o) of the liquid crystal.The power of the variable-focus lens 406 is therefore close to zero whenthe saturation voltage is applied, while the (negative) power of thevariable-focus lens 406 is at a maximum when no voltage is applied, as afunction of the difference between the two refractive indices (n1e andn2) and the curvature of the convex part 404. Intermediate voltages areused to select focus values between these extremes.

When the (negative) power of the variable-focus lens 406 is at a maximumthe two-way element 310 produces the diverging beam of FIG. 13A. Whenthe lens power is at a minimum the two-way element 310 produces theconverging beam of FIG. 13B.

The liquid crystal variable-focus lens 406 works in conjunction with alinear polarizer 410, which ensures that only light polarized parallelto the default director (FIG. 13A) passes into or out of the two-waydisplay element 310, i.e. only light focused by the variable-focus lens406.

As an alternative to using a single liquid crystal variable-focus lens406 in conjunction with a linear polarizer 410, two liquid crystalvariable-focus lenses with orthogonal rubbing directions can be used tofocus light of all polarizations [Berreman80].

The combined power of the fixed-focus positive lenses 400 and 412 isbalanced against the power of the variable-focus negative lens 406 toyield a focus range from short negative through to short positive, asillustrated in FIG. 13A and FIG. 13B respectively.

During two-way use of the element 310, display and capture may betime-multiplexed, with each frame period divided into a (relativelylonger) display interval and a (relatively shorter) capture interval,with the variable-focus lens 406 refocused appropriately before eachinterval.

As shown in FIGS. 14A and 14B, if the variable-focus lens 406 isn't fastenough to be refocused twice per frame then a pair of variable-focuslenses 406 and 416 with orthogonal rubbing directions can be used, onededicated to display focus and the other dedicated to capture focus. Inthis case a fast switchable polarization rotator 418 [Sharp00] can beused to selectively rotate the light polarization zero or ninetydegrees, and so select between display and capture focus.

FIG. 14A shows the first variable-focus lens 406 active to collimate thebeam 414 for display. FIG. 14B shows the second variable-focus lens 416active to collimate the beam 414 for capture. For clarity the figuresshow the unused variable-focus lens (406 or 416) made inoperative via anapplied saturation voltage. In practice, however, the unused lens isactually made inoperative by the polarization rotator 418, making thevoltage applied to it irrelevant.

Each light sensor 152 of the light sensor array 150 is preferably anactive pixel sensor (APS) [Fossum04] so that the entire array can beexposed simultaneously during the capture interval and then subsequentlyread out.

For color applications, each light emitter 182 of the light emitterarray 180 is preferably a full-color emitter such as a stack of red,green and blue OLEDs [Aziz10]; and each light sensor 152 may be afull-color sensor such as a sensor stack [Merrill05], or a sensor arraywith color filters. In addition, each light emitter 182 and light sensor152 may utilise any of the implementation options discussed in relationto the scanning light field display element below.

Each light emitter 182 and/or light sensor 152 may also supporttime-of-flight range-finding, as discussed in relation to the scanninglight field display element below.

The variable-focus lenses 406 and 416 are shown with inhomogeneous gaps,allowing the use of simple electrodes. Since the speed of liquid crystalrotation decreases with decreasing gap size, homogeneous gaps can beused to increase the speed of rotation, although this necessitates theuse of multi-segment electrodes [Lin11].

There are several disadvantages to using an array-based light fielddisplay element. Since each light emitter 182 is typically a diffuseemitter, only a portion of the generated light is actually emittedthrough the exit pupil of the display element. Since the size of theemitter array 180 is constrained by the spatial sampling period 120(since this constrains the width of the display element), the number ofangular samples may be overly constrained. And given practical limits onthe complexity of the lenses used to focus the output from the displayelement (and input to the two-way display element), it is difficult toachieve high off-axis beam quality.

These limitations are avoided in the scanning display element 210,scanning camera element 230, and scanning two-way display element 310described next.

Scanning Light Field Display Element

FIG. 15 shows a block diagram of a scanning embodiment of the lightfield display element 210 of the light field display 200.

The display element 210 scans an output beam of light 500 intwo-dimensional raster fashion across the 2D angular field 124, and foreach direction (a, b) modulates the beam to produce the desired radiance134 specified in the output light field view image 502, which is a viewimage 122 of a light field video 110.

Over the duration of a single pulse (described below) the beam 500corresponds to a particular output beam 212 in FIG. 7B, and to thereconstruction beam 192 in FIG. 5B.

The scanning display element 210 relies on the persistence of vision toinduce the perception of a continuous optical light field throughout theangular field of view 124.

The beam 500 is scanned in the line direction by fast line scanner 504(with an illustrative line rate of 100 kHz), and in the orthogonal(frame) direction by slow frame scanner 506 (with an illustrative framerate of 100 Hz).

The fast line scanner 504 and slow frame scanner 506 may be separate, ormay be combined in a 2D (biaxial) scanner.

The scanners are controlled by timing generator 510, which itself iscontrolled by an external frame sync signal 512, which is shared withother display elements 210. The frame scanner 506 is controlled by aframe sync signal 514 derived from the external frame sync signal 512,while the line scanner 504 is controlled by a line sync signal 516.

The radiance controller 520 controls the radiance of the output beam.Under the control of a sampling clock 518 from the timing generator 510,it reads the next radiance value 134 from the output view image 502 andgenerates a signal to control the radiance of the output beam.

If the angular scan velocity of the fast scanner 504 is angle-dependent(e.g. because the fast scanner is resonant) then the timing generator510 adjusts the sampling clock 518 accordingly to ensure a constantangular sampling period 126.

The beam generator 522 generates the light beam, and the radiancemodulator 524 modulates the radiance of the beam, typically in responseto a beam power signal from the radiance controller 520. Implementationchoices are described below.

The pulse duration should be matched to the angular sampling period 126to ensure proper reconstruction. If a shorter pulse (of correspondinglyhigher power) is used, then proper reconstruction can be effectedoptically, as described below in relation to FIG. 20.

As described earlier, the required beam power is obtained by multiplyingthe required radiance 134 by the 5D sampling period (i.e. the 1Dtemporal sampling period 114, the 2D spatial sampling period 120, andthe 2D angular sampling period 126), and dividing it by the pulseduration.

The pulse duration is obtained by dividing the angular sampling period126 by the angular scan velocity of the fast scanner 504. If the angularscan velocity is angle-dependent (e.g. because the fast scanner isresonant), then the pulse duration is also angle-dependent.

The scanned output beam 500 may be focused according to an output focussource 526. The output focus source 526 may comprise an array of focusvalues each associated with a beam direction, i.e. corresponding to thesampling focus 138 associated with the spectral radiance 128.Alternatively it may comprise a single focus value which may change fromone frame to the next (or at some other rate). Output focus controller528 retrieves the focus value (or the next focus value, controlled bythe sampling clock 518 from the timing generator 510), and generates asignal to control the focus of the output beam.

The output focus modulator 530 modulates the focus of the beam accordingto the signal from the output focus controller 528. Implementationchoices are described below. If the display 200 is only required tooperate in the fixed-focus far-field regime then the output focusmodulator 530 may impart fixed focus on the beam, i.e. it may consist ofa simple fixed-focus lens.

The display element 210 optionally incorporates multiple beam generators522 and radiance modulators 524 to generate multiple adjacent beams 500simultaneously.

Beam Generator

The beam generator 522 may be monochromatic, but is more usefullypolychromatic. FIG. 16 shows a block diagram of a polychromatic beamgenerator and radiance modulator assembly 540, which replaces the beamgenerator 522 and radiance modulator 524 of FIG. 15.

The polychromatic beam generator and radiance modulator assembly 540includes a red beam generator 542 and radiance modulator 544, a greenbeam generator 546 and radiance modulator 548, and a blue beam generator550 and radiance modulator 552. Each radiance modulator is responsive torespective signals from the radiance controller 520 shown in FIG. 15.The modulated red and green beams are combined via beam combiner 554.The resultant beam is combined with the modulated blue beam via beamcombiner 556. The beam combiners may be dichroic beam combiners capableof combining beams of different wavelengths with high efficiency. Tomaximise the reproducible gamut the red, green and blue beam generators542, 546 and 550 ideally have central wavelengths close to the primecolor wavelengths of 450 nm, 540 nm and 605 nm respectively [Brill98].

The beam generator 522 (or beam generators 542, 546 and 550) mayincorporate any suitable light emitter, including a laser [Svelto10],laser diode, light-emitting diode (LED), fluorescent lamp, andincandescent lamp. Unless the emitter is intrinsically narrowband (e.g.the emitter is a laser, laser diode, or LED), the beam generator mayincorporate a color filter (not shown). Unless the emitted light iscollimated, with adequately uniform power across the full beam width,the beam generator may include conventional collimating optics, beamexpansion optics, and/or beam-shaping optics (not shown).

The radiance modulator 524 may be intrinsic to the beam generator 522(or beam generators 542, 546 and 550). For example, the beam generatormay be a semiconductor laser which allows its power and pulse durationto be modulated directly by modulating its drive current.

If the radiance modulator 524 is distinct from the beam generator thenthe beam generator (or its light emitter) may be shared between a numberof display elements 310. For example, a number of display elements mayshare a lamp, or may share a single laser source via a holographic beamexpander [Shechter02, Simmonds11].

Each color light emitter may be particularly effectively implementedusing a semiconductor laser, such as a vertical-cavity surface-emittinglaser (VCSEL) [Lu09, Higuchi10, Kasahara11]. A VCSEL produces alow-divergence circular beam that at a minimum only requires beamexpansion.

Frequency-doubling via second harmonic generation (SHG) [Svelto10]provides an alternative to direct lasing at the target wavelength.

Radiance Modulator

If the radiance modulator 524 is distinct from the beam generator thenit may consist of any suitable high-speed light valve or modulator,including an acousto-optic modulator [Chang96, Saleh07], and anelectro-optic modulator [Maserjian89, Saleh07]. In the latter case itmay exploit the Franz-Keldysh effect or the quantum-confined Starkeffect to modulate absorption, or the Pockels effect or the Kerr effectto modulate refraction and hence deflection. The radiance modulator mayinclude optics (not shown) to manipulate the beam before and/or aftermodulation, i.e. to optimise the coupling of the beam and the modulator(e.g. if there is a mismatch between the practical aperture of themodulator and the width of the beam before and/or after the modulator).

If the modulation is binary then intermediate radiances may be selectedby temporally dithering the beam, i.e. pseudorandomly opening andclosing the valve throughout the nominal pulse duration with a dutycycle proportional to the required power. Dithering reduces artifacts inthe reconstructed light field.

For the illustrative display configuration the required radiancemodulation rate is 100 MHz (or an order of magnitude more if themodulation is binary). Both acousto-optic and electro-optic modulatorssupport this rate, as do modulators that are intrinsic to the beamgenerator.

Focus Modulator

The output focus modulator 530 may utilise any suitable variable-focuslens, including a liquid crystal lens [Berreman80, Kowel86, Naumov99,Lin11], a liquid lens [Berge07], a deformable membrane mirror[Nishio09], a deformable-membrane liquid-filled lens [Fang08], anaddressable lens stack [Love09], and an electro-optic lens (e.g.exploiting the Pockels effect or Kerr effect to modulate refraction)[Shibaguchi92, Saleh07, Jacob07, Imai11].

An addressable lens stack [Love09] consists of a stack of N birefringentlenses, each with a different power (e.g. half the power of itspredecessor), and each preceded by a fast polarization rotator (e.g.[Sharp00]). The 2̂N possible settings of the binary rotators yield acorresponding number of focus settings. For example, 10 lenses yield1024 focus settings.

Fast polarization rotators can also be used to select among a smallnumber of variable-focus lenses (as described in relation to FIGS. 14Aand 14B). Such a lens consists of a stack of N variable-focusbirefringent lenses, each preceded by a fast polarization rotator. Onepair of rotators is enabled at a time to select the variable-focus lensbracketed by the pair (the first rotator selects the lens; the secondrotator deselects subsequent lenses). This allows fast switching betweenvariable focus settings, even if the variable focus lenses themselvesare relatively slow. Each variable-focus lens in the stack can then bededicated to one display pass (which may be viewer-specific orscene-specific), and the rotators can be used to rapidly select theappropriate lens for each display pass in turn. The stack optionallyincludes an additional rotator after the final lens to allow the finalpolarization of the beam to be constant, e.g. if the optical pathcontains polarization-sensitive downstream components.

For the illustrative display configuration the required focus modulationrate is 100 MHz to support per-sample focus, a modest multiple of 100 Hzto support multiple single-focus display passes (e.g. for multipleviewers), and around 4 Hz to support single-viewer gaze-directed focus.All of the variable-focus lens technologies described above support a 4Hz focus modulation rate. Lens stacks utilising polarization rotatorssupport modulation rates in excess of 1 kHz. Electro-optic lensessupport modulation rates in excess of 100 MHz.

Line and Frame Scanners

The fast line scanner 504 and slow frame scanner 506 may each utiliseany suitable scanning or beam-steering mechanism, including a (micro-)electromechanical scanning mirror [Neukermans97, Gerhard00, Bernstein02,Yan06], an addressable deflector stack (‘digital light deflector’)[Titus99], an acousto-optic scanner [Vallese70, Kobayashi91, Saleh07],and an electro-optic scanner [Saleh07, Naganuma09, Nakamura10].

Most scanner technologies can support the 100 Hz illustrative framerate. Fast scanner technologies such as resonant microelectromechanicalscanners and electro-optic scanners can support the 100 kHz illustrativeline rate.

If the fast line scanner 504 is resonant then it may monitor (orotherwise determine) its own angular position and provide the timinggenerator 510 with angular position information to assist the timinggenerator with generating an accurate sampling clock 518.

Microelectromechanical scanners provide a particularly good combinationof scan frequency and angular field, and are described in more detaillater in this specification.

Scanning Light Field Camera Element

FIG. 17 shows a block diagram of a scanning embodiment of the lightfield camera element 230 of the light field camera 220.

The camera element 230 scans an input beam of light 600 intwo-dimensional raster fashion across the 2D angular field 124, and foreach direction (a, b) samples the beam to produce the desired radiance134 in the input light field view image 602, which is a view image 122of a light field video 110.

Over the duration of a single exposure (discussed below) the beam 600corresponds to a particular input beam 232 in FIG. 8B, and to thesampling beam 166 in FIG. 3B.

The beam 600 is scanned in the line direction by fast line scanner 504(with an illustrative line rate of 100 kHz), and in the orthogonal(frame) direction by slow frame scanner 506 (with an illustrative framerate of 100 Hz). Implementation choices for the scanners are asdescribed above in relation to the scanning display element 210.

The scanners are controlled by timing generator 510, which itself iscontrolled by an external frame sync signal 512, which is shared withother camera elements 230. The frame scanner 506 is controlled by aframe sync signal 514 derived from the external frame sync signal 512,while the line scanner 504 is controlled by a line sync signal 516.

The radiance sensor 604 senses the radiance of the beam, or, moretypically, a quantity representative of the radiance, such as beamenergy (i.e. beam power integrated over time). Implementation choicesare described below.

The radiance sampler 606, controlled by a sampling clock 518 from thetiming generator 510, samples the radiance-representative value (e.g.beam energy) from the radiance sensor 604, and converts it to a linearor non-linear (e.g. logarithmic) radiance value 134 which it writes tothe input view image 602. Implementation choices are described below.

As described earlier, the radiance 134 may be obtained by dividing asampled beam energy value by the 5D sample size (i.e. 1D exposureduration, 2D spatial sample size, and 2D angular sample size).

The nominal maximum sample exposure duration is obtained by dividing theangular sampling period 126 by the angular scan velocity of the fastscanner 504. If the angular scan velocity is angle-dependent (e.g.because the fast scanner is resonant), then the exposure duration isalso angle-dependent.

To improve the signal to noise ratio of the captured radiance 134, theeffective exposure duration can be increased beyond the nominal maximumexposure duration by using a sensor array as described below in relationto FIG. 23A and FIG. 23B.

To ensure proper band-limiting, the radiance sensor 604 nominally has anactive spatial extent that matches the angular sampling period 126.However, when coupled with the maximum sample exposure duration thisproduces blur in the fast scan direction. To avoid such blur, either theexposure duration needs to be reduced or the spatial extent of thesensor 604 in the fast scan direction needs to be reduced. The latterapproach can be realised by implementing the sensor 604 using a lineararray of narrow photodetectors, as also described below in relation toFIG. 23A and FIG. 23B.

The scanned input beam 600 may be focused according to an input focussource 608. The input focus source 608 may comprise an array of focusvalues each associated with a beam direction, i.e. corresponding to thesampling focus 138 associated with the spectral radiance 128.Alternatively it may comprise a single focus value which may change fromone frame to the next (or at some other rate). Input focus controller610 retrieves the focus value (or the next focus value, controlled bythe sampling clock 518 from the timing generator 510), and generates asignal to control the focus of the input beam.

The input focus modulator 612 modulates the focus of the beam accordingto the signal from the input focus controller 610. Implementationchoices for the input focus modulator 612 are the same as for the outputfocus modulator 530, as discussed above. If the camera 220 is onlyrequired to operate in the fixed-focus far-field regime then the inputfocus modulator 612 may impart fixed focus on the beam, i.e. it mayconsist of a simple fixed-focus lens

Radiance Sensor

The radiance sensor 604 may be monochromatic, but is more usefullypolychromatic. If polychromatic, it may utilize a stacked color sensor[Merrill05], or an array of sensors with color filters.

The sensor 604 may incorporate any suitable photodetector(s), includinga photodiode operating in photoconductive or photovoltaic mode, aphototransistor, and a photoresistor.

The sensor 604 may incorporate analog storage and exposure controlcircuitry [Fossum04].

Radiance Sampler

The radiance sampler 606 may incorporate any analog-to-digital converter(ADC) with suitable sampling rate and precision, typically with apipelined architecture [Levinson96, Bright00, Xiaobo10]. For theillustrative display configuration the sampling rate is 100 Msamples/sand the precision is 12 bits. The sampler 606 may incorporate multipleADCs to convert multiple color channels in parallel, or it maytime-multiplex conversion of multiple color channels through a singleADC. It may also utilise multiple ADCs to support a particular samplingrate.

The sampler 606 may incorporate a programmable gain amplifier (PGA) toallow the sensed value to be offset and scaled prior to conversion.

Conversion of the sensed value to a radiance 134 may be performed beforeor after analog-to-digital conversion.

Time-of-Flight Range Finding

The light field camera 220 is optionally configured to performtime-of-flight (ToF) range-finding [Kolb09]. The camera then includesone or more light emitters for illuminating the scene with ToF-codedlight. The ToF-coded light is reflected by the scene and is detected andconverted to a depth by each camera element 230 every sampling period.

The radiance sensor 604 and radiance sampler 606 may be configured toperform ToF range-finding by incorporating circuitry to measure thephase difference between the coding of the outgoing light and the codingof the incoming light [Kolb09, Oggier11].

When configured to perform ToF range-finding the sampler 606 writes anestimated depth 136 to the input view image 602 every sampling period.

The ToF-coded light is ideally invisible, e.g. near-infrared (NIR). Thesensor 604 may sense the ToF-coded light using a photodetector that isalso used for sensing visible light, or the sensor 604 may include adedicated photodetector for ToF-coded light.

An alternative to the camera providing one or more ToF-coded lightemitters, each camera element 230 may, if also configured as a displayelement 210 (see below), emit its own ToF-coded light. The beamgenerator 522 may incorporate a light emitter for ToF-coded light, suchas an NIR light emitter.

If necessary, face detection can be used to disable ToF range-findingfor any sample (x, y, a, b) that would transmit ToF light into an eye.

Scanning Two-Way Light Field Display Element

FIG. 18 shows a block diagram of a scanning two-way light field displayelement 310 of the two-way light field display 300. It combines thefunctions of the light field display element 210 and the light fieldcamera element 230 shown in FIG. 15 and FIG. 17 respectively.

In the scanning two-way light field display element 310, the linescanner 504, frame scanner 506 and the timing generator 510 are sharedbetween the display and camera functions of the element.

A beamsplitter 614 is used to separate the output and input opticalpaths. It may be any suitable beamsplitter, including a polarizingbeamsplitter (discussed further below), and a half-silvered (orpatterned) mirror.

In the scanning two-way light field display element 310 display andcapture occur simultaneously, except when the angular field 124 isvisibility-based (as discussed later in this specification) when it mayvary significantly between display and capture.

Optical Design of Scanning Two-Way Light Field Display Element

FIG. 19A shows a plan view of an optical design for the scanning two-waylight field display element 310. The traced rays show the output opticalpath in operation, i.e. the element is generating output beam 500. FIG.19B shows the corresponding front elevation.

The height of the two-way element is the spatial sampling period 120.The width of the two-way element 310 is approximately twice the spatialsampling period 120.

Where the optical design is illustrated with particular componentchoices, note that it could be implemented using other equivalentcomponents, such as discussed in previous sections. This includes theuse of reflecting components in place of transmitting components andvice versa.

The design goal for the output optical path is to generate the outputbeam 500 so that it properly reconstructs, for a given direction (a, b),the corresponding 4D slice of the (bandlimited) continuous light field.

A laser 700 is used to produce a collimated beam 500 with a width asclose as possible to the spatial sampling period 120. The beam may beexpanded and/or shaped (by additional components not shown) after beinggenerated by the laser 700. The laser 700 implements the beam generator522 described in previous sections.

An angular reconstruction filter 702 is used to induce spread in theoutput beam equal to the angular sampling period 126. The angularreconstruction filter 702 is discussed in more detail below, in relationto FIG. 20.

A variable-focus lens 704 is used to control the focus of the outputbeam. It implements the output focus modulator 530.

A beamsplitter 706 is used to split the output and input optical paths.It implements the beamsplitter 614.

A fixed mirror 708 deflects the output beam to a biaxial scanning mirror710, described in the next section. The scanning mirror 710 scans theoutput beam 500 across the angular field 124. It implements both theline scanner 504 and the frame scanner 506.

As an alternative, the biaxial scanning function may be implementedusing two separate uniaxial scanning mirrors. In this configuration thefixed mirror 708 is replaced by a fast uniaxial scanning mirror (whichimplements the line scanner 504), and biaxial scanning mirror 710 isreplaced by a relatively slower uniaxial scanning mirror (whichimplements the frame scanner 506).

FIG. 19A shows the biaxial scanning mirror 710, and hence output beam500, at three distinct angles, corresponding to the center and the twoextremes of the angular field 124.

The angular reconstruction filter 702 can be implemented using a(possibly elliptical) diffuser [Qi05], or using an array of lenslets 730as shown in FIG. 20. The purpose of the angular reconstruction filter isto induce spread in the output beam equal to the angular sampling period126, and the use of lenslets 730 allows the spread angle to be preciselycontrolled. Each lenslet 730 acts on the input beam 732 to produce afocused output beamlet 734. Since the input beam 732 is collimated, theinduced spread angle is the angle subtended by the diameter of thelenslet 730 at the focal point of the lenslet. In order to decouple theinduced spread from the beam focus induced by the downstreamvariable-focus lens 704, the focal point of the lenslet 730 is ideallyplaced on the first principal plane of the variable-focus lens 704 (atleast approximately).

The larger the number of lenslets 730, the more uniform the overalloutput beam, which is the sum of the individual beamlets 734. Thesmaller the diameter of each lenslet 730, the shorter its focal lengthneeds to be to induce the same spread angle, thus the smaller the gapbetween the angular reconstruction filter 702 and the variable-focuslens 704 needs to be. In practice the array of lenslets 730 may bemolded into the face of the variable-focus lens 704.

If the output pulse duration matches the angular sampling period 126(and scanning is continuous rather than discrete in the fast scandirection) then the output beam spread angle is already correct in thefast scan direction, and spread only needs to be induced in the slowscan direction. In this case each lenslet 730 may be a cylindrical lensoriented in a direction perpendicular to the slow scan direction.

FIG. 21A shows a plan view of the optical design for the two-way lightfield display element 310. The traced rays show the input optical pathin operation, i.e. the element is sampling input beam 600. FIG. 21Bshows the corresponding front elevation.

The design goal for the input optical path is to sample the input beam600 so that it properly filters, for a given direction (a, b), thecorresponding 4D slice of the continuous light field.

The biaxial scanning mirror 710 (or pair of uniaxial scanning mirrors)scans the input beam 600 across the angular field 124, as describedabove for the output optical path.

The fixed mirror 708 and beamsplitter 706 deflect the input beam tofixed mirror 712, which deflects the beam through variable-focus lens714.

The variable-focus lens 714 is used to control the focus of the inputbeam. It implements the input focus modulator 612.

The variable-focus lens 714 is followed by a fixed-focus lens 716, whichfocuses the (nominally collimated) input beam, via an aperture 718, ontoa photodetector 720. The photodetector 720 implements the radiancesensor 604.

For color sensing, the photodetector 720 may consist of a photodetectorstack [Merrill05], or a photodetector array with color filters.

The laser 700 may produce a substantially polarized beam (i.e. becauseit incorporates a polarizing Brewster window as its exit mirror), inwhich case it is efficient for the beamsplitter 706 to be polarizing,i.e. to split the outgoing and incoming beams based on polarization[vonGunten97]. Further, if the variable-focus lenses 704 and 714 arebirefringent (e.g. they are liquid-crystal lenses), they then only needto act on their respective beam polarization and are thus simplified.Even if the laser 700 does not intrinsically produce a highly polarizedbeam, it may incorporate or be followed by a polarizer for this purpose(not shown).

Biaxial Scanning Mirror

A uniaxial microelectromechanical (MEMS) scanner typically consists of amirror attached to a frame by a pair of perfectly elastic torsionalhinges, and is driven to rotate about the hinges via an electrostatic,magnetic or capacitive coupling between the mirror and a driver. In abiaxial MEMS scanner [Neukermans97], the inner frame holding the mirroris attached to a fixed outer frame via a further pair of hinges arrangedorthogonally to the mirror hinges, allowing the inner frame to be drivento rotate orthogonally to the mirror. The mirror is typically drivenresonantly while the inner frame is not.

In a typical biaxial MEMS scanner the inner and outer frames surroundthe mirror, and so the area of the mirror is a fraction of the footprintof the device. This makes such a device non-optimal for use in a lightfield display where the relative aperture of the scanner is important.This can be ameliorated by elevating the mirror above the scanningmechanism, as is the practice in digital micromirror devices (DMDs)[Hornbeck96, DiCarlo06].

FIG. 22A shows a plan view of an example biaxial MEMS scanner 710 withan elevated mirror, but otherwise of conventional design [Neukermans97,Gerhard00, Bernstein02, Yan06]. A central platform 740 is attached bytorsional hinges 742 to an inner frame 744. The inner frame 744 isattached by orthogonally-arranged torsional hinges 746 to a fixed outerframe 748. The central platform 740 is driven to rotate about the hinges744, while the inner frame 744 is driven to rotate in the orthogonaldirection about the hinges 746. A post 750, mounted on the platform 740,holds a mirror 752 (shown in outline) elevated above the scanningmechanism.

FIG. 22B shows a cross-sectional front elevation of the biaxial MEMSscanner 710, showing the mirror 752 elevated above the scanningmechanism by the post 750. The elevation of the mirror 752 above thescanning mechanism is chosen to accommodate the maximum scan angle.

FIG. 22B does not show the drive mechanisms, which may be of anyconventional design as described above. By way of example, the centralplatform 740 may incorporate a coil for conducting an alternatingcurrent, thus producing a time-varying magnetic field which interactswith the field of a permanent magnet below the platform (not shown) toproduce the required time-varying torque. Likewise, the inner frame 744may incorporate a coil whose field interacts with the field of apermanent magnet.

For present purposes, to support the illustrative line rate, the centralplatform 740 is driven resonantly [Turner05] and implements the fastline scanner 504, while the inner frame 744 is driven directly andimplements the slow frame scanner 506.

As previously mentioned, control logic associated with the scanner 710may monitor (or otherwise determine) the angular position of the centralplatform 740 in the resonant scan direction [Melville97, Champion12] forthe purposes of assisting the timing generator 510 with generating anaccurate sampling clock 518.

Extending Exposure Duration Using a Photodetector Array

The nominal exposure duration of a single light field sample during ascan is limited by the angular sampling period 126, and may therefore bevery short. However, it is possible to deploy a linear photodetectorarray parallel to the fast scan direction, in place of a singlephotodetector 720, to extend the exposure duration.

FIG. 21A, as described above, shows the scanning mirror 710 scanning themoving input beam 600 across the angular field 124. Equivalently, FIG.23A shows, via a simplified configuration which excludes extraneousoptical components, the scanning mirror 710 scanning a stationary beam760 corresponding to a fixed point source 224 across the photodetector,here replaced by a linear photodetector array 762 consisting of Mphotodetectors.

If samples are taken from the linear photodetector array 762 atprecisely the rate at which the stationary beam 760 is scanned acrossit, then M time-successive samples from the M photodetectors can besummed to yield a sample value with an effective exposure duration Mtimes longer than the nominal exposure duration.

As indicated in FIG. 23A, the linear photodetector array 762 covers anangular field M samples wide, representing M successive periods of thesampling clock 518. At a given time t these samples correspond to timesranging from t minus M/2 to t plus M/2, and M successive samples arebeing accumulated in parallel at any given time.

To avoid vignetting when using a linear photodetector array 762, theangular field 124 must be reduced by M times the angular sampling period126.

While sample readout and summation can be carried out using digitallogic, a relatively high sampling clock rate 518 (e.g. 100 MHz for theillustrative configuration) motivates an analog design.

To this end, FIG. 23B shows the photodetector array 762 consisting of ananalog photodetector array 764 coupled with an analog shift register768. Each period of the input sampling clock 518 the shift register 768is shifted up, and the value from each photodetector 766 is added to thecorresponding shift register stage 770. The value shifted into the first(bottom) shift register stage 770 is zero. The value shifted out of thelast (top) shift register stage 770 is converted, via ananalog-to-digital converter (ADC) 772, to a beam energy digital samplevalue 774. This in turn is converted to a radiance 134 as previouslydescribed. The ADC 772 may be any suitable ADC as previously described.

While the analog photodetector array 764 and the analog shift register768 may be distinct, in some practical implementations they can beclosely integrated. For example, if a bucket brigade device (BBD)[Sangster77, Patel78] is used as the analog shift register 768, thenphotodiodes 766 can be directly integrated into its storage nodes 770.And if a linear charge-coupled device (CCD) [Tompsett78] is used as theanalog photodetector array 764, it can intrinsically also be operated asan analog shift register 768.

The analog photodetector array 764 can also be implemented separatelyfrom the analog shift register 768, for example as a standard array ofactive pixel sensors (APSs) [Fossum04], and the analog shift registercan be implemented for example as a standard bucket brigade device(BBD), augmented with a third clock signal to control the transfer ofcharge from the photodetector array 764.

The effective exposure duration can be further increased by accumulatingsamples in the slow scan direction. This is achieved by deploying anarray of M′ linear photodetector arrays 762 to simultaneously capture M′adjacent lines of samples. During capture, M′ sample values 774 are thenproduced every period of the sampling clock 518, rather than just one,and each such sample 774 is added (once converted to a radiance) to itscorresponding radiance 134 in the input view image 602. The totalradiance 134 is scaled to the longer exposure duration by dividing it byM′.

For the illustrative display configuration, setting M=M′=100 (i.e. each1/10 of the angular field 124) yields an exposure duration of 100 us.

In addition to increasing the effective exposure duration, the linearphotodetector array 742 can be used to capture sharper samples byincorporating a multiple K of narrower photodetectors 746 (and shiftregister stages 770) per angular sampling period 126, and clocking theentire device the multiple K of the sampling clock 518. An additionalanalog storage node, inserted between the last shift register stage 770and the ADC 772, is then used to accumulate K successive analog samples,with the combined value being digitized and read out according to thesampling clock 518.

Just as the radiance sensor 604 (and hence the photodetector 720) may beconfigured for ToF range-finding, so may the photodetector array 762.For example, if ToF range-finding is based on phase measurement [Kolb09,Oggier11], then the photodetector array 762 may be configured toaccumulate phase samples in parallel.

Arrays of Two-Way Light Field Display Elements

FIG. 24 shows a simplified block diagram of an array of two-way lightfield display elements 310 operating in display mode. The 2D scanner 508represents both the 1D line scanner 504 and the 1D frame scanner 506.

FIG. 25A shows a plan view of the optical design of one row of a two-waylight field display 300, operating in display mode. The display consistsof an array of two-way light field display elements 310, each generatingan output beam 500. The array is shown at a single instant in time, witheach beam pointing in the same direction. Each beam has the same,slightly divergent, focus.

FIG. 25B shows a corresponding front elevation of the display 300.Successive display elements 310 are rotated 180 degrees to improve theuniformity of the output.

FIG. 25C shows the front elevation rotated 90 degrees.

For clarity, FIGS. 25A, 25B and 25C only show a small number of two-waydisplay elements 310. In practice a two-way light field display 300 cancontain any number of elements 310, e.g. numbering in the thousands ormillions. For the illustrative configuration it contains 125,000 displayelements.

FIG. 26 shows a plan view of one row of the display 300, rotated asshown in FIG. 25B, with each element 310 generating a beam 500corresponding to a single point source behind the display, hence atdifferent times during their scan cycles. The gaps in the output are dueto the double width of the display element 310 relative to the spatialsampling period 120.

FIG. 27 shows a plan view of one row of the display 300, rotated asshown in FIG. 25C, with each element 310 generating a beam 500corresponding to a single point source behind the display, hence atdifferent times during their scan cycles. The gaps in the output shownin FIG. 26 are now essentially eliminated because the display elements310 are rotated so that their width matches the spatial sampling period120.

FIG. 28 shows a plan view of one row of the display 300, rotated asshown in FIG. 25B, with each element 310 generating a beam 500corresponding to a single point source 204 in front of the display,hence at different times during their scan cycles. The gaps in theoutput are again due to the double width of the display element 310relative to the spatial sampling period 120.

FIG. 29 shows a plan view of one row of the display 300, rotated asshown in FIG. 25C, with each element 310 generating a beam 500corresponding to a single point source 204 in front of the display,hence at different times during their scan cycles. The gaps in theoutput shown in FIG. 28 are now essentially eliminated because thedisplay elements 310 are rotated so that their width matches the spatialsampling period 120.

FIG. 30 shows a simplified block diagram of an array of two-way lightfield display elements 310 operating in camera mode.

FIG. 31A shows a plan view of the optical design of one row of a two-waylight field display 300, operating in camera mode. The display consistsof an array of two-way light field display elements 310, each capturingan input beam 600. The array is shown at a single instant in time, witheach beam pointing in the same direction. Each beam has the same,slightly convergent, focus.

FIG. 31B shows a corresponding front elevation of the display 300.Successive display elements 310 are rotated 180 degrees to improve theuniformity of the input.

FIG. 31C shows the front elevation rotated 90 degrees.

FIG. 32 shows a plan view of one row of the display 300, rotated asshown in FIG. 31B, with each element 310 capturing a beam 600corresponding to a single point source 224 in front of the display,hence at different times during their scan cycles. The gaps in the inputare due to the double width of the display element 310 relative to thespatial sampling period 120.

FIG. 33 shows an plan view of one row of the display 300, rotated asshown in FIG. 31C, with each element 310 capturing a beam 600corresponding to a single point source 224 in front of the display,hence at different times during their scan cycles. The gaps in the inputshown in FIG. 32 are now essentially eliminated because the displayelements 310 are rotated so that their width matches the spatialsampling period 120.

Oscillating Display

As described in relation to FIG. 26, FIG. 28 and FIG. 32, the gaps inthe output and input are due to the double width of the display element310 relative to the spatial sampling period 120. This can be amelioratedby oscillating the array of two-way display elements 310 between twopositions that are a distance of one spatial sampling period 120 apart,and displaying and/or capturing half of a light field frame 116 at eachposition.

More generally, beyond displaying (or capturing) one half frame in oneof two positions, it is possible to display (or capture) 1/N frame inone of N positions, in either one spatial dimension or both spatialdimensions.

The angular field 124 of the display element 310 is, in general,constrained by the ratio of the beam width to the element width.Reducing the beam width relative to the element width allows for agreater angular field 124, but requires a higher value of N.

FIG. 34A shows a cross-sectional side elevation of a two-way light fielddisplay 300, adapted to oscillate the array of two-way display elements310 vertically.

The display 300 consists of a display panel 800, movably attached to achassis 802. The display panel 800 incorporates the array of two-waydisplay elements 310. A frame 804 is attached to the chassis 802,surrounding the panel 800 and holding a transparent cover glass 806 thatprotects the panel 800.

The display panel 800 is movably attached to the chassis 802 via a setsprings 808, each attached to a bracket 810 on the back of the panel 800and a matching bracket 812 on the chassis 802.

The display panel 800 is moved vertically via an actuator 814 driving arod 816. The rod is attached to a bracket 818 on the back of the panel800 and the actuator is attached to a matching bracket 820 on thechassis 802.

The actuator 814 may be any actuator suitable for displacing the weightof the panel 800 by the desired amount (e.g. 2 mm) at the desired rate(e.g. 100 Hz). For example, it may consist of current-carrying coilsacting on magnets embedded in the rod 816 [Petersen82, Hirabayashi95].

FIG. 34B shows the same cross-sectional side elevation of a two-waylight field display 300, but incorporating two contiguous display panels800 in the vertical dimension rather than just one.

FIG. 34C and FIG. 34D show the cross-sectional back elevationscorresponding to FIG. 34A and FIG. 34B respectively. FIG. 34D shows thedisplay 300 incorporating four contiguous display panels 800, two ineach dimension. This illustrates how a larger display 300 can beconstructed, in a modular fashion, from multiple smaller panels 800.

The oscillating display 300 is designed to oscillate its panel(s) 800,within one frame period (i.e. one temporal sampling period 114), betweentwo vertical positions that are a distance of one spatial samplingperiod 120 apart.

In one mode of operation the actuator 814 is used to directly determinethe vertical offset of the panel 800. The panel 800 is then moved asquickly as possible from one extreme vertical offset to the other, andthe next half-frame is displayed (or captured) as soon as the panel 800is in position. The display duty cycle is then a function of the speedof the actuator. The faster the actuator the higher the duty cycle. Thismode is illustrated by the graph of vertical offset versus time in FIG.35A.

In an alternative mode of operation the spring constants of the springs808 are chosen so that they and the panel 800 form a harmonic oscillatorwith the desired frequency. The actuator 814 is then used to drive theoscillator with the desired amplitude. This requires a less powerfulactuator than direct driving, and consumes less power during operation.

The disadvantage of harmonic oscillation is that the display 800 followsthe sinusoidal path shown in FIG. 35B and is therefore only momentarilystationary at the extreme vertical offsets. A compromise then needs tobe made between duty cycle and vertical motion blur. The lower the dutycycle the lower the blur, although, beneficially, the blur decreasesmore rapidly than the duty cycle due to the sinusoid. By way of example,FIG. 35B shows a duty cycle of 67%, corresponding to vertical motion of50%, i.e. a motion blur diameter of 25%.

If the oscillation is harmonic and the display element 310 is scanningthen the fast scan direction is ideally aligned with the oscillationaxis to minimise interaction between the oscillation and the scan.

The frequency of the harmonic oscillator is proportional to the squareroot of the ratio of the spring constant of the springs 808 to the massof the panel 800. Since both spring constants and masses are additive,the frequency is independent of the number of panels 800 used to createthe display 300.

As an alternative to using oscillation to merge two half-frame lightfields produced by a single display, the light fields produced by twodisplays can be combined via a beam combiner (e.g. a half-silvered glassplate).

Real-Time Capture and Display of a Light Field

In one important use-case, as illustrated in FIG. 11 and FIG. 12 anddescribed above, a light field display 200 receives and displays a lightfield from a (possibly remote) light field camera 220 in real time.

As discussed above, how capture focus is managed depends in part on theavailable focus modulation rate.

FIG. 36 shows an activity diagram for the display controller 342 and thecamera controller 340 cooperatively controlling focus based on theposition of the viewer (and optionally the viewer's gaze direction).

The display controller 342 periodically detects the face and eyes of theviewer (at 900) (or of each of several viewers), optionally alsoestimates the viewer's gaze direction (at 902), and transmits (at 904)the positions of the eyes (and optionally the gaze direction) to thecamera controller 340.

The camera controller 340 receives the eye positions (and optionally thegaze direction), and autofocuses accordingly (at 906). Autofocus mayrely on explicitly setting focus based on a depth obtained byrange-finding (discussed above), or on a traditional autofocus techniquesuch as phase detection between images from adjacent camera elements230, adaptively adjusting focus to maximise image sharpness in thedesired direction, or a combination of the two.

If the camera controller 340 only receives eye positions then it mayinfer a pair of possible gaze directions for each camera element 230based on the positions of the eyes. This implements the position-basedviewer-specific focus mode described earlier in relation to FIG. 10B. Ifthe camera controller 340 receives an estimate of the gaze directionthen it may use this directly. This implements the gaze-directedviewer-specific focus mode described earlier in relation to FIG. 10C andFIG. 10D.

If the camera supports per-sample autofocus then this is most naturallybased on the per-sample depth 136, and neither the eye positions nor theestimated gaze direction are required. If the camera supports per-frame(or per-sub-frame) focus modulation then autofocus can be based on theestimated or inferred gaze directions.

As previously discussed, if the positions of the eyes are used to inferpossible gaze directions for each camera element 230, then a separatedisplay pass (and hence capture pass) is ideally used for each eye.

In general, since autofocus may span multiple frames, when there aremultiple capture passes (e.g. corresponding to multiple viewers oreyes), autofocus context must be preserved over several frames for eachpass.

FIG. 37 shows an activity diagram for the display controller 342 and thecamera controller 340 cooperatively controlling focus based on thefixation point (or fixation depth) of the viewer. This again implementsthe gaze-directed viewer-specific focus mode described earlier inrelation to FIG. 10C and FIG. 10D.

The display controller 342 periodically detects the face and eyes of theviewer (at 900) (or of each of several viewers), estimates the viewer'sfixation point (or depth) (at 908), and transmits (at 910) the positionsof the eyes and the fixation point (or depth) to the camera controller340. The display controller 342 may estimate the fixation point (ordepth) based on the viewer's gaze direction in conjunction with thesample depth 136 in the incoming light field video 110, or on thevergence of the user's eyes, or on a combination of the two.

FIG. 38 shows an activity diagram for camera controller 340 and displaycontroller 342 cooperatively capturing and displaying a sequence oflight field frames 116 in real time.

The camera controller 340 periodically captures a light field frame (at920) and transmits it (at 922) to the display controller 342. Thedisplay controller 342 receives and optionally resamples the light fieldframe (at 924), and finally displays the light field frame (at 926).Resampling is discussed further below.

The resampling step 924 optionally uses a locally-captured light fieldframe to virtually illuminate the scene represented by theremotely-captured light field frame. This is straightforward via raytracing (discussed below) if the remotely-captured light field frame 116contains depth 136.

Display of a Previously-Captured Light Field Video

In another important use-case, a two-way light field display 300displays a previously-captured light field video.

FIG. 39 shows an activity diagram for two-way display controller 322displaying a light field video 110.

The diagram shows two parallel activities: a face-detection activity onthe left and a display activity on the right.

The face detection activity periodically detects the face and eyes ofthe viewer (at 900) (or of each of several viewers), stores the eyepositions in a datastore 930, estimates the viewer's fixation point (ordepth) (at 908), and stores the fixation point (or depth) in a datastore932. The controller estimates the fixation point (or depth) based on theviewer's gaze direction in conjunction with the sample depth 136 in thesource light field video 110 (stored in a datastore 934), or on thevergence of the user's eyes, or on a combination of the two.

The display activity periodically displays (at 926) the next light fieldframe 116 of the light field video 110. It optionally resamples (at 936)the light field prior to display, in particular to match the focus tothe estimated fixation plane. This again implements the gaze-directedviewer-specific focus mode described earlier in relation to FIG. 10C andFIG. 10D.

The display activity optionally captures (at 920) a light field frame116, allowing the subsequent resampling step (at 936) to use thecaptured light field frame to virtually illuminate the scene representedby the light field video. This is straightforward via ray tracing(discussed below) if the light field video 110 contains depth 136. Itallows real ambient lighting incident on the display 300 to light thescene in the video, and it allows the real objects visible to thetwo-way display (including the viewer) to be reflected by virtualobjects in the virtual scene.

The two parallel activities are asynchronous and typically havedifferent periods. For example, the face-detection activity may run at10 Hz while the display activity may run at 100 Hz. The two activitiescommunicate via the shared datastores.

Display of Light Field Video from a 3D Animation Model

In yet another important use-case, a two-way light field display 300generates and displays light field video from a 3D animation model.

FIG. 40 shows an activity diagram for two-way display controller 322generating and displaying light field video 110 from a 3D animationmodel.

The diagram shows two parallel activities: a face-detection activity onthe left and a display activity on the right.

The face detection activity periodically detects the face and eyes ofthe viewer (at 900) (or of each of several viewers), stores the eyepositions in a datastore 930, estimates the viewer's fixation point (ordepth) (at 908), and stores the fixation point (or depth) in a datastore932. The controller estimates the fixation point (or depth) based on theviewer's gaze direction in conjunction with depth information determinedfrom the 3D animation model (stored in a datastore 938), or on thevergence of the user's eyes, or on a combination of the two.

The display activity periodically renders (at 940) and displays (at 926)the next light field frame 116 from the 3D animation model. Duringrendering it matches the focus to the estimated fixation plane. Thisagain implements the gaze-directed viewer-specific focus mode describedearlier in relation to FIG. 10C and FIG. 10D.

Rendering a light field frame 116 is straightforward via ray tracing[Levoy96, Levoy00]. As illustrated in FIG. 3B, each spectral radiance128 may be generated by tracing, from a corresponding (now virtual)light sensor 152, a set of rays that sample the sampling beam 166, anddetermining the interaction of each ray with the 3D model [Glassner89].The rays are ideally chosen to sample the 4D sampling beam 166stochastically, to avoid low-frequency artifacts associated with regularsampling. Ray density may also be matched adaptively to scene complexityto reduce aliasing.

The two parallel activities are asynchronous and typically havedifferent periods. For example, the face-detection activity may run at10 Hz while the display activity may run at 100 Hz. The two activitiescommunicate via the shared datastores.

Although the rendering step 940 is shown performed by the two-waydisplay controller 322, it may also be performed by a separate computingdevice in communication with the two-way display controller 322.

The display activity optionally captures (at 920) a light field frame116, allowing the subsequent rendering step (at 940) to use the capturedlight field frame to virtually illuminate the scene represented by the3D animation model. This is again straightforward during ray tracing. Itallows real ambient lighting incident on the display 300 to light thevirtual scene, and it allows the real objects visible to the two-waydisplay (including the viewer) to be reflected by virtual objects in thevirtual scene.

The viewer's gaze can be reflected at each virtual surface it encountersto obtain the actual fixation point 262 (as shown in FIG. 10C). Thefixation point can then either be virtual or real, i.e. behind thedisplay or in front of the display respectively. If the fixation pointis virtual then the depth of the fixation point is determined by tracingthe gaze, via further reflections (if any), to an element 310. If thefixation point is virtual then the capture beam is diverging; if realthen the capture beam is converging. This allows the viewer to fixate ona real object via a reflection in a virtual object.

In addition to including light field video 110 captured by the two-waydisplay 300, the 3D animation model can include already-captured or livelight field video from other sources. This includes light field video110 from another two-way light field display 300 mounted back-to-backwith the present two-way light field display 300, allowing virtualobjects to overlay (and refract, when transparent) real objects visibleto the back-facing two-way display 300.

Distribution of Functions

The functions of the display controller 342 may be performed by adedicated controller associated with or embedded in the display 200, orby a separate device (or devices) in communication with the display 200.

Likewise, the functions of the camera controller 340 may be performed bya dedicated controller associated with or embedded in the camera 220, orby a separate device (or devices) in communication with the camera 220.

Light Field Resampling

Prior to display, a light field 110 may need to be resampled. This isnecessary if the temporal sampling period 114, spatial sampling period120 or angular sampling period 126 of the target display 200 differsfrom the corresponding sampling period of the source light field 110; iftheir respective spectral sampling bases 132 differ; if their respectivesampling focuses 138 differ; or if their respective light fieldboundaries 102 differ, e.g. one is rotated or translated relative to theother, or they have different curved shapes.

Translation may include translation in the z direction, e.g. to displayvirtual objects in front of the display.

In addition to spectral resampling, spectral remapping may be used tomap non-visible wavelengths (such as ultraviolet and near infrared) tovisible wavelengths.

Resampling is not required if the captured (or synthesised) light field110 being displayed matches the characteristics of the target lightfield display 200. For example, no resampling is required, by default,when pairs of identical two-way displays 300 are used together, e.g.each displaying the light field 110 captured by the other as shown inFIG. 11. However, resampling to translate the light field boundary of alight field video 110 to compensate for the spatial separation of a pairof back-to-back displays 300 can be used to implement practicalinvisibility for the region between the two displays.

Light field resampling involves generating, from an input light fieldvideo 110, a resampled output light field video 110. If the temporalsampling regime is unchanged, then it involves generating, from an inputlight field frame 116, a resampled output light field frame 116, i.e. aset of output light field view images 122, each corresponding to aposition (xy) on the spatial sampling grid of the output light fieldframe 116. One of the most common uses of light fields is to generatenovel 2D views [Levoy96, Levoy00, Isaksen00, Ng05a]. Resampling a lightfield equates to generating a set of novel 2D views.

As illustrated in FIG. 3B, each spectral radiance 128 has acorresponding (virtual) light sensor 152 and sampling beam 166.Computing a resampled output spectral radiance 128 involves identifyingall sampling beams 166 associated with the input light field frame 116that impinge on the light sensor 152 corresponding to the outputspectral radiance, and computing the weighted sum of each beam'scorresponding input spectral radiance 128. Each weigh is chosen to be(at least approximately) proportional to the overlap between the beamand the light sensor 152.

Additional Display Modes

The primary display mode of the light field display 200 is toreconstruct a continuous light field from a discrete light field 110representing a scene containing objects at arbitrary depths.

In addition to this primary display mode it is useful to support adisplay mode in which the display 200 emulates a conventional 2Ddisplay. Given a 2D image, this can be achieved in two ways. In thefirst approach the 2D source image is simply embedded at a convenientvirtual location in 3D, and the corresponding discrete light field isrendered and displayed. In this case the 2D image is limited to lying infront of or behind the display 200, subject to the minimum (negative orpositive) focal length and angular field 124 of the display elements210. The sample count of the 2D source image is then limited by theangular sample count of the display 200.

In the second approach the entire light field view image 122 of eachdisplay element 210 is set to a constant value equal to the value of thespatially-corresponding pixel in the 2D source image, and the displayelement focus is set to its minimum (negative or positive). The samplecount of the 2D source image is then limited by the spatial sample countof the display 200.

It is also useful to support a display mode where the scene is locatedat infinity. In this case the output of the display 200 is collimated,the view image 122 displayed by each display element 210 is identical,and the output focus is set to infinity. The required sample count ofthe collimated source image equals the angular sample count of thedisplay 200.

A collimated source image can be captured using a light field camera 220by focusing its camera elements 230 at infinity and either choosing oneview image 122 as the collimated image, or, for a superior image,averaging a number of view images 122 from a number of camera elements230 (and in the limit, from all of the camera elements 230). Theaveraged image is superior because it has a better signal-to-noiseratio, and because it better suppresses scene content not located atinfinity. This averaging approach represents a specific example of amore general synthetic aperture approach.

Synthetic Aperture

During capture, the light field view images 122 captured by any numberof adjacent camera elements 230 can be averaged to simulate the effectof a larger camera aperture [Wilburn05]. In this process, spectralradiances 128 that correspond to the same virtual point source 224 (asshown in FIG. 8B) are averaged. This may require view image resamplingto ensure alignment with the 4D sampling grid of the combined viewimage.

The use of a synthetic aperture results in a greater effective exposure,and therefore an improved signal to noise ratio, but shallower depth offield.

Staggered Element Timing

During capture (and subsequent display), the timing of the frame syncsignal used by different camera elements 230 (and display elements 210)can be stochastically staggered to provide more uniform sampling in thetime domain [Wilburn11]. This results in a smoother perception ofmovement when the light field video 110 is displayed, but with increasedmotion blur if a synthetic aperture is used.

Mirror Mode

The two-way light field display 300 can also be configured to act as amirror, i.e. where the captured light field is re-displayed in realtime. Capture and display focus is managed as described above.

In the simplest mirror mode each two-way element re-displays its owncaptured view image. This can operate via a sample buffer, a line bufferor a full view image buffer per element.

Image processing can also be performed on the light field betweencapture and re-display, e.g. image enhancement, relighting, and spectralremapping.

Audio

The light field display 200 can be configured to reproduce multiplechannels of digital audio associated with a light field video 110 byincluding digital-to-analog converters (DACs), amplifiers, andelectro-acoustic transducers (speakers) mounted along the periphery of(or otherwise in the vicinity of) the display.

The light field camera 220 can be configured to capture multiplechannels of digital audio as part of a light field video 110 byincluding a set acoustic sensors (microphones) mounted along theperiphery (or otherwise in the vicinity of) of the display, andanalog-to-digital converters (ADCs). A microphone may also beincorporated in each camera element 230.

Each audio channel may be tagged with the physical offset of themicrophone used to capture it to allow phased-array processing of theaudio [VanVeen88, Tashev08], e.g. for reducing ambient noise orisolating individual remote speakers [Anguera07] (e.g. after selectionvia gaze).

Phased-array techniques may also be used to focus the reproduction of aselected audio source (such as a remote speaker) at the local viewer whohas selected the source [Mizoguchi04] (e.g. after selection via gaze).This allows multiple viewers to attend to different audio sources withreduced interference.

A sufficiently dense array of speakers (e.g. with a period of 5 cm orless) may be used to reproduce an acoustic wave field [deVries99,Spors08, Vetterli09], allowing audio to be virtually localised to itsvarious sources, independent of the position of the viewer (i.e.listener). This ensures that aural perception of a displayed scene isconsistent with its visual perception. A correspondingly dense array ofmicrophones can be used to capture a real acoustic wave field, and anacoustic wave field is readily synthesized from a 3D animation modelcontaining audio sources.

The light field video 110 can thus be extended to include a time-varyingdiscrete acoustic wave field, i.e. consisting of a dense array of audiochannels.

A one-dimensional speaker array may be used to reproduce an acousticwave field in one dimension, e.g. corresponding to the horizontal planeoccupied by viewers of the display 200. A two-dimensional speaker arraymay be used to reproduce an acoustic wave field in two dimensions.

Two-Way Display Controller Architecture

FIG. 41 shows a block diagram of the two-way display controller 322,discussed earlier in relation to FIG. 11 and FIG. 12.

The display controller 342 should be considered equivalent to thetwo-way display controller 322 operating in display mode, and viceversa. The camera controller 340 should be considered equivalent to thetwo-way display controller operating in camera mode, and vice versa.

The two-way display controller 322 includes a two-way panel controller950 which coordinates the display and capture functions of a singletwo-way display panel 800. When a two-way display 300 incorporatesmultiple panels 800 they can be controlled in modular fashion bymultiple panel controllers 950.

The display and capture functions of each individual two-way displayelement 310 is controlled by a corresponding two-way element controller952. The element controller 952 utilises a view image datastore 954,which holds an output view image 502 for display and a captured inputview image 602 (as described earlier in relation to FIG. 15, FIG. 17 andFIG. 18).

During display, the display element 310 reads successive radiancesamples 134 from the output view image 502, while at the same time thepanel controller 950 writes new radiance samples 134 to the output viewimage 502. The view image datastore 954 only needs to accommodate afractional output view image 502 if reading and writing are wellsynchronised.

During capture, the panel controller 950 reads successive radiancesamples 134 from the input view image 602, while at the same time thedisplay element 310 writes new radiance samples 134 to the input viewimage 602. The view image datastore 954 only needs to accommodate afractional input view image 602 if reading and writing are wellsynchronised.

For the illustrative display configuration, the display has a totalmemory requirement of 6E11 bytes (600 GB) each for display and capture,assuming full (rather than fractional) view images.

The element controller 952 supports two display modes: standard lightfield display (from the output view image 502 in the view imagedatastore 954), and constant-color display (from a constant-colorregister).

The two-way display element controller block 956, consisting of thetwo-way element controller 952 and its view image datastore 954, isreplicated for each two-way display element 310.

The panel controller 950 and/or element controllers 952 may beconfigured to perform light field decompression prior to or duringdisplay and light field compression during or after capture. Light fieldinterchange formats and compression are discussed further below.

Each of the panel controller 950 and element controllers 952 maycomprise one or more general-purpose programmable processing units withassociated instruction and data memory, one or more graphics processingunits with associated instruction and data memory [Moreton05], andpurpose-specific logic such as audio processing, image/video processingand compression/decompression logic [Hamadani98], all with sufficientprocessing power and throughput to support a particular two-way displayconfiguration.

Although FIG. 41 shows one element controller 952 per display element310, an element controller 952 may be configured to control multipledisplay elements 310.

The panel controller 950 utilises a 2D image datastore 958 to hold a 2Dimage for display. As described earlier, the 2D image may be displayedby configuring each display element 310 to display a constant color. Inthis mode the panel controller 950 writes each pixel of the 2D image tothe constant-color register of the corresponding element controller 952.Alternatively, the 2D image may be displayed by synthesising a lightfield frame 116. In this mode the panel controller 950 synthesises alight frame 116, using a specified 3D location and orientation for the2D image, and writes each resultant output view image 122 to itscorresponding view image datastore 954.

The panel controller 950 utilises a collimated view image datastore 960when operating in collimated mode, holding a collimated output viewimage and a collimated input view image. As described earlier, incollimated display mode each display element 310 displays the sameoutput view image 122. The panel controller 950 can either broadcast thecollimated output view image to the element controllers 952 duringdisplay, or the collimated output view image can be written to theindividual view image datastores 954 prior to display.

As also described earlier, in collimated capture mode the collimatedoutput view image may be obtained by averaging a number of input viewimages 602. The panel controller 950 can perform this averaging duringor after capture.

A network interface 962 allows the panel controller 950 to exchangeconfiguration data and light field video 110 with external devices, andmay comprise a number of conventional network interfaces to provide thenecessary throughput to support light field video 110. For example, itmay comprise multiple 10 Gbps or 100 Gbps Gigabit Ethernet (GbE)interfaces, coupled to fiber or wire.

An input video interface 964 allows an external device to writestandard-format video to the display 300 for 2D display via the 2Ddatastore 958, allowing the display 300 to be used as a conventional 2Ddisplay.

When the display 300 is operating in collimated display mode, the inputvideo interface 964 also allows an external device to write collimatedlight field video 110 as standard-format video to the display fordisplay via the collimated view image datastore 960.

When the display 300 is operating in collimated capture mode, an outputvideo interface 966 allows other devices to read collimated light fieldvideo 110 from the display as standard-format video. This allowscollimated light field video 110 to be easily exchanged between a pairof two-way light field displays 300 using a pair of standard videointerconnections.

A display timing generator 968 generates the global frame sync signal512 used to control both display and capture (as described in relationto FIG. 15 and FIG. 17 respectively).

If the display is designed to oscillate, as described in relation toFIGS. 24A through 24D, a panel motion controller 970 drives the actuator814 and monitors the position of the piston 816.

The various components of the two-way display controller 322 communicatevia a high-speed data bus 972. Although various data transfers aredescribed above as being performed by the panel controller 950, inpractice they may be initiated by the panel controller (or othercomponents) but performed by DMA logic (not shown). The data bus 972 maycomprise multiple buses.

Although the various datastores are shown as distinct, they may beimplemented as fixed-size or variable-size regions of one or more memoryarrays.

Light Field Interchange Formats and Compression

While light field video 110 may be exchanged between compatible devices(including light field cameras 220, light field displays 200, and otherdevices) in uncompressed form, the throughput (and memory) requirementsof light field video typically motivate the use of compression. Theillustrative display configuration has a throughput of 4E13 samples/s(5E14 bits/s; 500×100 GbE links), and requires a frame memory of 6E11bytes (600 GB).

Compression may exploit the full 5D redundancy within time intervals ofa light field video 110 (i.e. including inter-view redundancy[Chang06]), or 4D redundancy within a light field frame 116 [Levoy96,Levoy00, Girod03]. It may also utilise conventional image or videocompression techniques on each (time-varying) light field view image122, such as embodied in the various JPEG and MPEG standards. 100:1compression based on 4D redundancy is typical [Levoy96, Levoy00].

Stereoscopic and multiview video utilised by 3D TV and video (3DV)systems contains a small number of sparse views, and H.264/MPEG-4 (viaits multiview video coding (MVC) profiles) supports 5D compression withthe addition of inter-view prediction to the usual spatial and temporalprediction of traditional single-view video [Vetro11]. MVC 5Dcompression can be applied to a dense light field video 110.

When the optional light field depth 136 is available, depth-basedcompression techniques can be used. Depth-based representations used in3DV systems include multiview video plus depth (MVD), surface-basedgeometric representations (e.g. textured meshes), and volumetricrepresentations (e.g. point clouds) [Alatan07, Muller11].

With MVD, the use of depth information allows effective inter-viewprediction from a sparser set of views than standard inter-viewprediction (i.e. MVC without depth), thus MVD allows a dense set ofviews to be more effectively synthesized from a sparse set of views,thus at least partly decoupling the view density of the interchangeformat from the view density of the display [Muller11].

By supporting 3DV formats the display 300 also becomes capable ofexchanging 3D video streams with other 3DV devices and systems.

Visibility-Based Two-Way Display Controller Architecture

As shown in FIG. 42A, each two-way display element 310 has an angularfield 980 (corresponding to the light field angular field 124), only asmall subset 982 of which is seen by the eye 240 of a viewer.

It is therefore efficient to only capture, transmit, resample, renderand display the subset 982 of each element's field (suitably expanded toallow for eye movement between frames), as this reduces the requiredcommunication and processing bandwidth, as well as the required power.This selective capture, processing and display relies on face detection.

If the two-way display element 310 is a scanning element, then thescanning time in one or both scanning directions can be reduced if thescan is limited to the visible field 982.

Assuming a minimum viewing distance of 200 mm and a visible field 982 10mm wide at the eye, the (one-way) throughput of the illustrative displayconfiguration (per viewer) is reduced by two orders of magnitude to 4E11samples/s (5E12 bits/s; 46×100 GbE links uncompressed; 1×100 GbE linkwith 46:1 compression), and the memory requirements to 6E9 bytes (6 GB).

As further shown in FIG. 42B, only a small number of display elements210 intersect a projection 984 of the foveal region of the retina of theeye 240. It is therefore efficient to capture, transmit, resample,render and display the light field using a reduced angular sampling rateoutside this region (suitably expanded to allow for eye movement betweenframes). This selective capture, processing and display relies on gazeestimation.

FIG. 43 shows a block diagram of the two-way display controller 322optimised for visibility-based display and capture.

Each full-field view image 122 (stored in the view image datastore 954of FIG. 41) is replaced by a smaller partial view image 122 (stored inthe partial view image datastore 986 in FIG. 43). Each partial viewimage only covers the corresponding element's eye-specific partialangular field 982 (shown in FIG. 41A).

The maximum required size of a partial view image is a function of theminimum supported viewing distance.

If the display 300 supports multiple viewers in viewer-specific mode(e.g. via multiple display passes), then the capacity of the partialview image datastore 986 can be increased accordingly. At a minimum, tosupport a single viewer during display and a single viewer duringcapture, the partial view image datastore 986 has a capacity of fourpartial view images, i.e. one per viewer eye 240.

Further, as discussed above in relation to FIG. 42B, each partial viewimage may be subsampled, and then replaced by a non-subsampled partialview image when the corresponding display element 310 falls within theprojection of the fovea. This can allow a further order of magnitudereduction in the size of each partial view image. In this approach anumber of non-subsampled partial view images are stored in a partialfoveal view image datastore 988, and each display element 310 within theprojection of the fovea is configured to use a designated partial fovealview image (in the datastore 988) in place of its own subsampled partialview image (in the datastore 986).

The maximum required number of foveal view images is a function of themaximum viewing distance at which foveal display is supported.

Assuming a maximum viewing distance of 5000 mm for foveal viewing, and afoveal field 984 of 2 degrees, the (one-way) throughput of theillustrative display configuration (per viewer) is reduced by a furtherfactor of six to 7E10 samples/s (8E11 bits/s; 8×100 GbE linksuncompressed; 1×100 GbE link with 8:1 compression; 1×10 GbE link with80:1 compression), and the memory requirements to 1E9 bytes (1 GB).

When the foveal regions of multiple viewers are non-overlapping, it ispossible to support viewer-specific focus within each viewer's fovealregion during a single display pass.

Visibility-based capture works in the same way, with the distinctionthat while visibility-based display is responsive to the position orgaze of one or more local viewers of the display, visibility-basedcapture is responsive to the position or gaze of one or more viewersviewing the captured light field on a remote display.

With visibility-based subsampling the element controller 952 supportstwo additional display modes: display with interpolation of radiancesamples 134 (from the subsampled output view image in the partial viewimage datastore 986), and foveal display (from the designated partialfoveal output view image in the partial foveal image datastore 988).

REFERENCES

The contents of the following publications, referred to within thisspecification, are herein incorporated by reference.

-   [Anguera07] X. Anguera et al, “Acoustic Beamforming for Speaker    Diarization of Meetings”, IEEE Transactions on Audio, Speech, and    Language Processing, 15(7), September 2007-   [Alatan07] A. A. Alatan et al, “Scene Representation Technologies    for 3DTV—A Survey”, IEEE Transactions on Circuits and Systems for    Video Technology, 17(11), November 2007-   [Amir03] A. Amir et al, “Calibration-Free Eye Gaze Tracking”, U.S.    Pat. No. 6,578,962, Jun. 17, 2003-   [Aziz10] H. Aziz and J. A. Coggan, “Stacked OLED Structure”, U.S.    Pat. No. 7,750,561, Jul. 6, 2010-   [Adelson91] E. H. Adelson and J. R. Bergen, “The Plenoptic Function    and the Elements of Early Vision”, Computational Models of Visual    Processing, M. Landy and J. A. Movshon (eds), MIT Press, 1991, pp.    3-20-   [Balogh06] T. Balogh, “Method and Apparatus for Displaying 3D    Images”, U.S. Pat. No. 6,999,071, Feb. 14, 2006-   [Barabas11] J. Barabas et al, “Diffraction Specific Coherent    Panoramagrams of Real Scenes”, Proceedings of SPIE, Vol 7957, 2011-   [Benzie07] P. Benzie et al, “A Survey of 3DTV Displays: Techniques    and Technologies”, IEEE Transactions on Circuits and Systems for    Video Technology, 17(11), November 2007-   [Berge07] B. Berge and J. Peseux, “Lens with Variable Focus”, U.S.    Pat. No. RE39,874, Oct. 9, 2007-   [Bernstein02] J. Bernstein, “Multi-Axis Magnetically Actuated    Device”, U.S. Pat. No. 6,388,789, May 14, 2002-   [Berreman80] D. W. Berreman, “Variable Focus Liquid Crystal Lens    System”, U.S. Pat. No. 4,190,330, Feb. 26, 1980-   [Betke00] M. Betke et al, “Active Detection of Eye Scleras in Real    Time”, IEEE CVPR Workshop on Human Modeling, Analysis and Synthesis,    2000-   [Brill98] M. H. Brill et al, “Prime Colors and Color Imaging”, The    Sixth Color Imaging Conference: Color Science, Systems and    Applications, Nov. 17-20, 1998, IS&T-   [Bright00] W. J. Bright, “Pipeline Analog-to-Digital Conversion    System Using Double Sampling and Method of Operation”, U.S. Pat. No.    6,166,675, Dec. 26, 2000-   [Champion12] M. Champion et al, “Scanning Mirror Position    Determination”, U.S. Pat. No. 8,173,947, May 8, 2012-   [Chang96] I-C. Chang, “Acousto-Optic Bragg Cell”, U.S. Pat. No.    5,576,880, Nov. 19, 1996-   [Chang06] C-L. Chang et al, “Light Field Compression Using    Disparity-Compensated Lifting and Shape Adaptation”, IEEE    Transactions on Image Processing, 15(4), April 2006-   [Connor11] R. A. Connor, “Holovision™ 3D Imaging with Rotating    Light-Emitting Members”, U.S. Pat. No. 7,978,407, Jul. 12, 2011-   [deVries99] D. de Vries and M. M. Boone, “Wave Field Synthesis and    Analysis Using Array Technology”, Proceedings of the 1999 IEEE    Workshop on Applications of Signal Processing to Audio and    Acoustics, 17-20 Oct. 1999-   [DiCarlo06] A. DiCarlo et al, “Yokeless Hidden Hinge Digital    Micromirror Device”, U.S. Pat. No. 7,011,415, Mar. 14, 2006-   [Duchowski07] A. T. Duchowski, Eye Tracking Methodology—Theory and    Practice, Second Edition, Springer 2007-   [Fossum04] E. R. Fossum et al, “Active Pixel Sensor with Intra-Pixel    Charge Transfer”, U.S. Pat. No. 6,744,068, Jun. 1, 2004-   [Fang08] J. Fang et al, “Wide-Angle Variable Focal Length Lens    System”, U.S. Pat. No. 7,359,124, Apr. 15, 2008-   [Georgiev06] T. Georgiev et al, “Spatio-Angular Resolution Tradeoff    in Integral Photography”, Eurographics Symposium on Rendering, 2006-   [Georgiev09] T. Georgiev et al, “Plenoptic Camera”, U.S. Pat. No.    7,620,309, Nov. 17, 2009-   [Gerhard00] G. J. Gerhard et al, “Scanned Display with Pinch, Timing    and Distortion Correction”, U.S. Pat. No. 6,140,979, Oct. 31, 2000-   [Girod03] B. Girod et al, “Light Field Compression Using    Disparity-Compensated Lifting”, 2003 IEEE International Conference    on Multimedia and Expo, July 2003-   [Glassner89] A. S. Glassner (ed.), An Introduction to Ray Tracing,    Academic Press, 1989-   [Hamadani98] M. Hamadani and R-S. Kao, “MPEG Encoding and Decoding    System for Multimedia Applications”, U.S. Pat. No. 5,845,083, Dec.    1, 1998-   [Hansen10] D. W. Hansen and Q. Ji, “In the Eye of the Beholder: A    Survey of Models for Eyes and Gaze”, IEEE Transactions on Pattern    Analysis and Machine Intelligence, 32(3), March 2010-   [Hartridge22] H. Hartridge, “Visual Acuity and the Resolving Power    of the Eye”, The Journal of Physiology, 57(1-2), Dec. 22, 1922-   [Higuchi10] Y. Higuchi and K. Omae, “Method of Manufacturing Nitride    Semiconductor Light Emitting Element and Nitride Semiconductor Light    Emitting Element”, US Patent Application Publication No.    2010/0098127, Apr. 22, 2010-   [Hirabayashi95] Y. Hirabayashi et al, “Moving Magnet-Type Actuator”,    U.S. Pat. No. 5,434,549, Jul. 18, 1995-   [Hoffman08] D. M. Hoffman et al, “Vergence-Accommodation Conflicts    Hinder Visual Performance and Cause Visual Fatigue”, Journal of    Vision, 8(3):33, 28 Mar. 2008-   [Hornbeck96] L. J. Hornbeck, “Active Yoke Hidden Hinge Digital    Micromirror Device”, U.S. Pat. No. 5,535,047, Jul. 9, 1996-   [Huai 1] G. Hua et al, “Face Recognition Using Discriminatively    Trained Orthogonal Tensor Projections”, U.S. Pat. No. 7,936,906, May    3, 2011-   [Imai11] T. Imai et al, “Variable-Focal Length Lens”, U.S. Pat. No.    8,014,061, Sep. 6, 2011-   [Isaksen00] A. Isaksen et al, “Dynamically Reparameterized Light    Fields”, Proceedings of the 27^(th) Annual Conference on Computer    Graphics and Interactive Techniques, ACM, 2000-   [Jacob07] S. A. Jacob, “Variable Focal Length Electro-Optic Lens”,    US Patent Application Publication No. 2007/0070491, Mar. 29, 2007-   [Jones03] M. J. Jones and P. Viola, “Fast Multi-View Face    Detection”, Mitsubishi Electric Research Laboratories TR2003-96,    August 2003-   [Jones06] M. J. Jones and P. Viola, “Method and System for Object    Detection in Digital Images”, U.S. Pat. No. 7,099,510, Aug. 29, 2006-   [Kasahara11] D. Kasahara et al, “Demonstration of Blue and Green    GaN-Based Vertical-Cavity Surface-Emitting Lasers by Current    Injection at Room Temperature”, Applied Physics Express, 4(7), July    2011-   [Kolb09] A. Kolb et al, “Time-of-Flight Sensors in Computer    Graphics”, Eurographics Conference 2009-   [Kobayashi91] K. Kobayashi, “Laser Beam Scanning System”, U.S. Pat.    No. 4,992,858, Feb. 12, 1991-   [Koike08] T. Koike et al, “Three-Dimensional Display Device”, US    Patent Application Publication No. 2008/0036759, Feb. 14, 2008-   [Kowel86] G. T. Kowel et al, “Adaptive Liquid Crystal Lens”, U.S.    Pat. No. 4,572,616, Feb. 25, 1986-   [Lazaros08] N. Lazaros et al, “Review of Stereo Vision Algorithms:    from Software to Hardware”, International Journal of    Optomechatronics, 2, 2008-   [Levinson96] R. A. Levinson and S. T. Ngo, “Pipelined Analog to    Digital Converter”, U.S. Pat. No. 5,572,212, Nov. 5, 1996-   [Levoy96] M. Levoy and P. Hanrahan, “Light Field Rendering”,    Proceedings of the 23rd Annual Conference on Computer Graphics and    Interactive Techniques, ACM, 1996-   [Levoy00] M. Levoy and P. Hanrahan, “Method and System for Light    Field Rendering”, U.S. Pat. No. 6,097,394, Aug. 1, 2000-   [Lin11] H-C. Lin et al, “A Review of Electrically Tunable Focusing    Liquid Crystal Lenses”, Transactions on Electrical and Electronic    Materials, 12(6), 25 Dec. 2011-   [Lienhart03] R. Lienhart et al, “A Detector Tree of Boosted    Classifiers for Real-Time Object Detection and Tracking”,    Proceedings of the 2003 International Conference on Multimedia and    Expo—Volume 1, 2003-   [Love09] G. D. Love et al, “High-Speed Switchable Lens Enables the    Development of a Volumetric Stereoscopic Display”, Optics Express    17(18), 31 Aug. 2009-   [Lu09] T-C. Lu et al, “Development of GaN-Based Vertical cavity    Surface-Emitting Lasers”, IEEE Journal of Selected Topics in Quantum    Electronics, 15(3), May/June 2009-   [Maserjian89] J. Maserjian, “Multiple Quantum Well Optical    Modulator”, U.S. Pat. No. 4,818,079, Apr. 4, 1989-   [Melville97] C. D. Melville, “Position Detection of Mechanical    Resonant Scanner Mirror”, U.S. Pat. No. 5,694,237-   [Merrill05] R. B. Merrill and R. A. Martin, “Vertical Color Filter    Sensor Group and Semiconductor Integrated Circuit Fabrication Method    for Fabricating Same”, U.S. Pat. No. 6,894,265, May 17, 2005-   [Mizoguchi04] H. Mizoguchi et al, “Visually Steerable Sound Beam    Forming System Based On Face Tracking and Speaker Array”,    Proceedings of the 17^(th) International Conference on Pattern    Recognition, 23-26 Aug. 2004-   [Moreton05] H. P. Moreton et al, “User Programmable Geometry    Engine”, U.S. Pat. No. 6,900,810, May 31, 2005-   [Muller11] K. Muller et al, “3-D Video Representation using Depth    Maps”, Proceedings of the IEEE, 99(4), April 2011-   [Naganuma09] K. Naganuma et al, “High-Resolution KTN Optical Beam    Scanner”, NTT Technical Review 7(12), 12 Dec. 2009-   [Nakamura10] K. Nakamura et al, “Electrooptic Device”, U.S. Pat. No.    7,764,302, Jul. 27, 2010-   [Naumov99] A. F. Naumov et al, “Control Optimization of Spherical    Modal Liquid Crystal Lenses”, Optics Express 4(9), April 1999-   [Neukermans97] A. P. Neukermans and T. G. Slater, “Micromachined    Torsional Scanner”, U.S. Pat. No. 5,629,790, May 13, 1997-   [Ng05a] R. Ng et al, “Light Field Photography with a Hand-held    Plenoptic Camera”, Stanford Tech Report CTSR 2005-02, 2005-   [Ng05b] R. Ng, “Fourier Slice Photography”, ACM Transactions on    Graphics, 24(3), July 2005-   [Nishio09] M. Nishio and K. Murakami, “Deformable Mirror”, U.S. Pat.    No. 7,474,455, Jan. 6, 2009-   [Oggier11] T. Oggier et al, “On-Chip Time-Based Digital Conversion    of Pixel Outputs”, U.S. Pat. No. 7,889,257, Feb. 15, 2011-   [Ong93] E. Ong et al, “Static Accommodation in Congenital    Nystagmus”, Investigative Ophthalmology & Visual Science, 34(1),    January 1993-   [Palmer99] S. E. Palmer, Vision Science—Photons to Phenomenology,    MIT Press, 1999-   [Patel78] M. P. Patel, “Bucket Brigade Circuit”, U.S. Pat. No.    4,130,766, Dec. 19, 1978-   [Petersen82] C. C. Petersen, “Linear Motor”, U.S. Pat. No.    4,363,980, Dec. 14, 1982-   [Plainis05] S. Plainis et al, “The Effect of Ocular Aberrations on    Steady-State Errors of Accommodative Response”, Journal of Vision,    5(5):7, 23 May 2005-   [Qi05] J. Qi et al, “Tailored Elliptical Holographic Diffusers for    LCD Applications”, Journal of the SID, 13(9), 2005-   [Saleh07] B. E. A. Saleh and M. C. Teich, Fundamentals of Photonics    (Second Edition), Wiley, 2007-   [Sangster77] F. L. J. Sangster, “Charge Transfer Device”, U.S. Pat.    No. 4,001,862, Jan. 4, 1977-   [Schwerdtner06] A. Schwerdtner et al, “Device for Holographic    Reconstruction of Three-Dimensional Scenes”, US Patent Application    Publication No. 2006/0250671, Nov. 9, 2006-   [Seitz06] S. M. Seitz et al, “A Comparison and Evaluation of    Multi-View Stereo Reconstruction Algorithms”, IEEE Computer Society    Conference on Computer Vision and Pattern Recognition, 17-22 Jun.,    2006-   [Shechter02] R. Shechter et al, “Compact Beam Expander with Linear    Gratings”, Applied Optics 41(7), 1 Mar. 2002-   [Sharp00] G. D. Sharp, “Switchable Achromatic Polarization Rotator”,    U.S. Pat. No. 6,097,461, Aug. 1, 2000-   [Shibaguchi92] T. Shibaguchi, “Electrooptic Device”, U.S. Pat. No.    5,140,454, Aug. 18, 1992-   [Shih00] S-W. Shih and J. Liu, “A Calibration-Free Gaze Tracking    Technique”, Proceedings of the 15th International Conference on    Pattern Recognition, 3-7 Sep. 2000-   [Simmonds11] M. D. Simmonds and R. K. Howard, “Projection Display”,    U.S. Pat. No. 7,907,342, Mar. 15, 2011-   [Spors08] S. Spors et al, “The Theory of Wave Field Synthesis    Revisited”, 124th Convention of the Audio Engineering Society, 17-20    May 2008-   [Svelto10] 0. Svelto, Principles of Lasers (Fifth Edition),    Springer, 2010-   [Szeliski99] R. S. Szeliski and P. Golland, “Method for Performing    Stereo Matching to Recover Depths, Colors and Opacities of Surface    Elements”, U.S. Pat. No. 5,917,937, Jun. 29, 1999-   [Tashev08] I. Tashev and H. Malvar, “System and Method for    Beamforming Using a Microphone Array”, U.S. Pat. No. 7,415,117, Aug.    19, 2008-   [Titus99] C. M. Titus et al, “Efficient, Accurate Liquid Crystal    Digital Light Deflector”, Proc. SPIE Vol. 3633, Diffractive and    Holographic Technologies, Systems, and Spatial Light Modulators VI,    June 1999-   [Tompsett78] M. F. Tompsett, “Charge Transfer Imaging Devices”, U.S.    Pat. No. 4,085,456, Apr. 18, 1978-   [Turk92] M. Turk and A. P. Pentland, “Face Recognition System”, U.S.    Pat. No. 5,164,992, Nov. 17, 1992-   [Turner05] A. M. Turner et al, “Pulse Drive of Resonant MEMS    Devices”, U.S. Pat. No. 6,965,177, Nov. 15, 2005-   [Urey11] H, Urey and M. Sayinta, “Apparatus for Displaying 3D    Images”, US Patent Application Publication No. 2011/0001804, Jan. 6,    2011-   [Vallese70] L. M. Vallese, “Light Scanning Device”, U.S. Pat. No.    3,502,879, Mar. 24, 1970-   [VanVeen88] B. D. Van Veen and K. M. Buckley, “Beamforming: A    Versatile Approach to Spatial Filtering”, IEEE ASSP Magazine, April    1988-   [Vetro11] A. Vetro et al, “Overview of the Stereo and Multiview    Video Coding Extensions of the H.264/MPEG-4 AVC Standard”,    Proceedings of the IEEE, 99(4), April 2011-   [Vetterli09] M. Vetterli and F. P. C. Pinto, “Audio Wave Field    Encoding”, US Patent Application Publication No. 2009/0248425, Oct.    1, 2009-   [vonGunten97] M. K. von Gunten and P. Bevis, “Broad Band Polarizing    Beam Splitter”, U.S. Pat. No. 5,625,491, Apr. 29, 1997-   [Watanabe96] M. Watanabe, “Minimal Operator Set for Passive Depth    from Defocus”, Proceedings of the IEEE Computer Society Conference    on Computer Vision and Pattern Recognition, 18-20 Jun., 1996-   [Wilburn05] B. Wilburn et al, “High Performance Imaging Using Large    Camera Arrays”, ACM Transactions on Graphics—Proceedings of ACM    SIGGRAPH 2005-   [Wilburn11] B. Wilburn et al, “Apparatus and Method for Capturing a    Scene Using Staggered Triggering of Dense Camera Arrays”, U.S. Pat.    No. 8,027,531, Sep. 27, 2011-   [Xiaobo10] C. Xiaobo et al, “A 12 bit 100 MS/s pipelined analog to    digital converter without calibration”, Journal of Semiconductors    31(11), November 2010-   [Yan06] J. Yan et al, “MEMS Scanner with Dual Magnetic and    Capacitive Drive”, U.S. Pat. No. 7,071,594, Jul. 4, 2006-   [Yaras10] F. Yaras et al, “State of the Art in Holographic    Displays”, Journal of Display Technology, 6(10), October 2010

1. A virtual window system, the system comprising a light field cameradevice and a light field display device arranged back-to-back, the lightfield camera device configured to capture an input light field videostream, the light field display device configured to display an outputlight field video stream based on the input light field video stream. 2.The system of claim 1, wherein the light field camera device comprisesat least one input focus modulator, whereby the focus of at least partof the input light field video stream is modulated according to afixation depth of a viewer of the light field display device.
 3. Thesystem of claim 1, wherein the light field display device comprises atleast one output focus modulator, whereby the focus of at least part ofthe output light field video stream is modulated according to a fixationdepth of a viewer of the light field display device.
 4. The system ofclaim 1, wherein the light field camera device comprises at least oneinput focus modulator and the light field display device comprises atleast one output focus modulator, whereby the focus of at least part ofthe input light field video stream and the focus of at least part of theoutput light field video stream are modulated, respectively, accordingto a fixation depth of a viewer of the light field display device. 5.The system of claim 1, wherein the input light field video stream isresampled to translate its light field boundary to compensate for theseparation of the light field camera device and the light field displaydevice, to produce the output light field video stream.
 6. The system ofclaim 1, wherein the input light field video stream is incorporated intoa rendered light field video stream to produce the output light fieldvideo stream, the rendered light field video stream comprising at leastone virtual 3D object.
 7. The system of claim 6, wherein the renderedlight field video stream comprises at least one virtual 3D object whichinteracts with part of the input light field video stream, theinteraction selected from the group comprising: reflection andrefraction.
 8. The system of claim 1, wherein the light field cameradevice comprises an array of light field camera elements, each cameraelement comprising: (a) an array of radiance sensors for sensing theradiance of a set of input beams over time, the input beams spanning atwo-dimensional angular field; and (b) an input focus modulator formodulating the focus of the input beams over time.
 9. The system ofclaim 1, wherein the light field camera device comprises an array oflight field camera elements, each camera element comprising: (a) ascanner for scanning an input beam across a two-dimensional angularfield; (b) an input focus modulator for modulating the focus of theinput beam over time; and (c) a radiance sensor for sensing the radianceof the input beam over time.
 10. The system of claim 1, wherein thelight field display device comprises an array of light field displayelements, each display element comprising: (a) an array of beamgenerators and radiance modulators for generating and modulating theradiance of a set of output beams over time, the output beams spanning atwo-dimensional angular field; and (b) an output focus modulator formodulating the focus of the output beams over time.
 11. The system ofclaim 1, wherein the light field display device comprises an array oflight field display elements, each display element comprising: (a) abeam generator for generating an output beam of light; (b) a radiancemodulator for modulating the radiance of the output beam over time; (c)an output focus modulator for modulating the focus of the output beamover time; and (d) a scanner for scanning the output beam across atwo-dimensional angular field.
 12. A virtual window system, the systemcomprising two two-way light field camera and display devices arrangedback-to-back, each device configured to capture an input light fieldvideo stream and to display an output light field video stream based onthe input light field video stream from the other device.
 13. The systemof claim 12, wherein at least one of the two-way devices comprises atleast one input focus modulator, whereby the focus of at least part ofits input light field video stream is modulated according to a fixationdepth of a viewer of the other two-way device.
 14. The system of claim12, wherein at least one of the two-way devices comprises at least oneoutput focus modulator, whereby the focus of at least part of its outputlight field video stream is modulated according to a fixation depth of aviewer of the two-way device.
 15. The system of claim 12, wherein atleast one of the two-way devices comprises at least one input focusmodulator and the other two-way device comprises at least one outputfocus modulator, whereby the focus of at least part of its input lightfield video stream and the focus of at least part of the output lightfield video stream of the other device are modulated, respectively,according to a fixation depth of a viewer of the other two-way device.16. The system of claim 12, wherein at least one of the input lightfield video streams is resampled to translate its light field boundaryto compensate for the separation of the two-way devices, to produce thecorresponding output light field video stream.
 17. The system of claim12, wherein at least one of the input light field video streams isincorporated into a rendered light field video stream to produce thecorresponding output light field video stream, the rendered light fieldvideo stream comprising at least one virtual 3D object.
 18. The systemof claim 17, wherein the rendered light field video stream comprises atleast one virtual 3D object which interacts with part of thecorresponding input light field video stream, the interaction selectedfrom the group comprising: reflection and refraction.
 19. The system ofclaim 12, wherein at least one of the two-way devices comprises an arrayof two-way camera and display elements, each two-way element comprising:(a) an array of radiance sensors for sensing the radiance of acorresponding set of input beams over time, the input beams spanning atwo-dimensional angular field; (b) an input focus modulator formodulating the focus of the input beams over time; (c) an array of beamgenerators and radiance modulators for generating and modulating theradiance of a set of output beams over time, the output beams spanning atwo-dimensional angular field; and (d) an output focus modulator formodulating the focus of the output beams over time.
 20. The system ofclaim 12, wherein at least one of the two-way devices comprises an arrayof two-way camera and display elements, each two-way element comprising:(a) a scanner for scanning an input beam and an output beam across atwo-dimensional angular field; (b) an input focus modulator formodulating the focus of the input beam over time; (c) a radiance sensorfor sensing the radiance of the input beam over time; (d) a beamgenerator for generating the output beam of light; (e) a radiancemodulator for modulating the radiance of the output beam over time; and(f) an output focus modulator for modulating the focus of the outputbeam over time.